Category Archives: Linux

An OpenSprinkler success story

I wanted to automate the watering system at home preferably using open-source and DIY systems. The initial plan was to go with plain RPi, OpenHAB and some GPIO code driving the sprinkler valves but the problem was creating a useful UI to control the system since OpenHAB is too clunky and generic looking. I was also not quite ready diving deep into embedded programming and OpenHAB programming model. OpenSprinkler  seemed to have everything I needed, a RPi hat with all the correct electrical outputs and an open source firmware and android app I could modify myself if needed. In the end, programming the sequences myself and trying to make a decent UI would be just too much work for a small pet project so I went with a ready solution.

The requirements

  1. Three separate zones around the house, max 7 sprinklers per zone.
  2. Each zone must be turned on separately due to the pressure requirement for the sprinklers to work.
  3. Pump that drives the water must be turned on automatically with each zone valve.

Setting up the OSPi

OpenSprinkler offers fully assembled systems but I decided to go the DYI route using my own RPi and just buying the OSPi hat.

  • RPi 4 Model B 2GB
  • RPi official charger
  • OSPi (VAC)
  • 32GB SD card
  • Orbit 57056 2-Pin European Transformer

Finding a 24VAC power supply with EU plug was quite a challenge, the listed model from Orbit was one of the rare ones I could find online (on Amazon).

It is mentioned in OpenSprinkler documentation that a separate power supply for RPi is recommended. This was confirmed while testing where I saw dmesg errors about voltage not being sufficient and RPi rebooting endlessly. I ended up using the official RPi charger and the 24VAC charger at the same time.

Installing raspbian and OSPi firmware was easy with no problems encountered. Assembling the OSPi was also not problematic, other than drilling some holes into the supplied enclosure for the USB cable and WiFi adapter.

The WiFi

The built in WiFi on RPi would not work even half of the required distance and was simply horrendous. Onboard WiFi can be disabled by modifying /boot/config.txt and adding

dtoverlay=disable-wifi
in the [all] section.

After checking compatibility lists and reviews for RPi compatible USB WiFi adapters I went with Edimax EW-7811UN. I disabled the integrated card and configured the /boot/wpa_supplicant.conf to connect to the dedicated WiFi extender AP as a priority.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
 ssid="Home_Ext"
 psk="pass"
 id_str="ext1"
 priority=5
}


network={
 ssid="Home"
 psk="pass"
 id_str="main"
 priority=10
}

In the end I managed to achieve a not so great but stable signal from the house to controller box at around -60db. For the WiFi extender I went super cheap using TP-LINK TL-WR840N (15EUR) and positioning it with no walls blocking the signal other than a single garage door. I also added a small script to sudo cron to automatically restart RPi in case of any network downtime.

ping -c4 192.168.1.1 > /dev/null

if [ $? != 0 ]
then
  sudo /sbin/shutdown -r now
else
  echo $(date) "Internet is UP"
fi

Driving the Pump

Looking for a relay to turn the pump on and off I decided to go with an off delay relay as an extra safety that will automatically turn off after the selected period of time. This is just in case OSPi goes haywire and does not turn off as scheduled or someone makes a mistake of turning the sprinklers on for too long. Pump draining all the water and running dry is a very bad scenario I would like to avoid. The model is a Tracon multifunction relay AC/DC 12-240V driven by 24VAC OSPi.

Putting it together

Relay is connected to OSPi port 0 (master zone) which is always turned on with either valve 1, 2 or 3. Relay drives the first power socket for the pump. The other two sockets are for Orbit 24VAC and RPi charger. This way the pump can be disconnected at any time and used manually.

The valves

24VAC valves are quite common. I found three candidates from Orbit, Rainbird and Cleber. In the end it came down to price and availability, so I went with 3x Rainbird CP075 off eBay, roughly 30$ each.

Finally, to connect the valves to OSPi I got some 4×0.75mm cable and some waterproof clips to connect them on the valve side. These are automatic clips put in a box full of gel which seals it when closed.

Operation and conclusion

It turns out the OSPi firmware and app has the exact functions I need to drive the setup. Master zone translates perfectly into the pump relay. For each valve, the "continuous" setting (which is default) allows you to setup a single schedule program and OSPi will automatically drive each valve one after another and not all at once (which would not work due to low pressure). Without the continuous setting one would have to write a separate program for each valve which is a bit clunky.

One thing that does not work quite as good is automatic rain delay. The idea is, if sprinklers are scheduled to work today but there is a rain forecast, delay the program for some time, like a day. Unfortunately, if it does not rain at all, delay is still present. It would appear that OSPi only checks the forecasts but does not adjust the delay according to actual mm of rain that has fallen. I need to research this function in more depth to figure out the exact behavior and whether I can improve upon it.

Another glitch that appears once a month or so is that OSPi is randomly not accessible. This is fixed with the main router reboot. I am not sure yet what exactly causes the problem, whether the auto-reboot script works.. more investigation is needed. It probably boils down to not so great WiFi connection.

In the end I am quite happy I went with the OpenSprinkler and not a full DIY solution. It saved time, does everything I require and I am able to modify it if ever needed.

 

2023 update

After the system being dormant through winter of 2022/2023, the RPi would no longer boot. The Sandisk Ultra SD card seems to have got corrupted for some reason so I had to re-image and reconfigure OSPI again. I replaced the card with a Sandisk Industrial series card which are supposedly more tolerant to heat and cold. We'll see how long this one lasts.

Other than that the system is still working great.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Apache http to https redirect – use 307

Who knew that a simple thing like HTTP redirects would be so complicated? It turns out clients will just change POST to GET on 301 (Postman, curl, everyone?), same with 302 which really behaves like 303 and that is also an old implementation "bug". Yeah, seriously.

If you have a REST API with POST (or other non-GET) request endpoints (who doesn't?) this behaviour will completely destroy everything.  Many guides (top google results) out there for configuring Apache redirect do not mention this problem. The code of choice would be 308 Permanent Redirect but that is fairly new so I would not risk it, older clients don't know it exists. The only thing left is 307 which does not allow changing methods on redirect – exactly how it should be.

Solution:

<VirtualHost *:80>
    ServerName example.com
    Redirect 307 / https://example.com/
</VirtualHost>

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Setting env variables with hyphen and running a program

Docker compose allows you very unrestrictive naming of your environment variables. It allows you to use hyphen and other "special" characters in variables names. When you need to use these variables in regular shell you are out of luck, bash and many other shells do not allow hyphens in variable names. But this is merely a shell restriction, so how to do it?

With env

env -i 'TZ=Europe/Berlin' \
'PORT=8080' \
'BASE-URL=http://localhost:8080' \
'DB[0]_CONNECTION-URL=jdbc:postgresql://localhost:5432/postgres' \
'DB[0]_USERNAME=username' \
'DB[0]_PASSWORD=password' java -jar myapp.jar

Note that env ignores all inherited env variables so you might need to redefine them:

env -i JAVA_HOME=$JAVA_HOME \
'TZ=Europe/Berlin' \ 
'PORT=8080' java -jar myapp.jar

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Updating server from Debian Stretch to Buster

Not the most pleasant experience.. I expected a smoother upgrade from Debian team. Upgrading from 8 to 9 was a walk in the park compared to this.

1. MySQL silently fails to start after upgrade

MySQL was left behind at version 5.5 after upgrade and would just not start anymore, probably segfaulting. There is no mysql-server package anymore so I had really no other option but to remove it and install mariadb. In addition, I had trouble running mariadb due to requirement to run mysql_upgrade .. but I couldn't run that because I had no working instance of mysql server running! Installing package default-mysql-server instead somehow solved the problem.

2. phpMyAdmin removed from packages

Not sure how maintaining phpMyAdmin is such a big task that the package was dropped from repos. Regular setup is simply unzipping the code and add an apache config.

3. docker fails due to nftables switch

Docker is such a big and important package these days… and breaks due to iptables no longer being the default. I would expect the upgrade process to not do the switch in this case.

4. apt autoremove anomaly

For some reason running autoremove wanted to purge essential packages such as php, gcc and python3. I did not pay too much attention but alerts started going off when ifup was getting removed and my ssh connection was lost. ?????????? (10 question marks)

 

Luckily this was all of the troubles, dealing with broken wordpress plugins was a relaxing task afterwards.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Creating a new torrent and seeding with Transmission

You have setup a Transmission server on your Linux box together with Transmisison Web or something along those lines and now you are wondering.. how can I actually seed a NEW file?

I couldn't find a straightforward answer on the web so here is the short tutorial:

  1. Upload your file to your transmission download directory
  2. cd to that directory and create a torrent file (lets say the file you uploaded was called  myfile.rar):
    transmission-create -o myfile.torrent -c "this is my file comment" -t tracker1 -t tracker2 -t tracker3 myfile.rar

    Replace tracker1, tracker3, tracker3, …trackerN with a bunch of trackers. Better specify more than one in case they go down. Here is a cool little list of public trackers.

  3. Download the new .torrent file you just created, open Transmission Web and add the torrent. Since the file already exists in download directory, Transmisison will just revalidate the data and start seeding. *mind blown*
  4. Distribute the torrent file to your people or generate a magnet link with
    transmission-show -m myfile.torrent

     

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Ubuntu 18.04 on MacBook Pro 11.5 – a sad state of affairs

There was an extra MacBook Pro 11.5 lurking around so I decided to install Ubuntu 18.04 on it and try to setup a usable workstation.

A culmination of several issues prompted me to not pursue this setup further. Linux drivers and MacBook hardware just don't play along very well.

Display flickering/corruption on main display

The bottom part of the HiDPI screen is experiencing some kind of flickering as tracked by this bug. Changing desktop environment, distros and X server configuration did not result in any improvements. For a moment Wayland seemed to have solved the issue only to reappear on next boot.

Since I also connected 3 external monitors this was not a deal breaker. External monitors did not display this issue.

Fan going at 100% most of the time

Even at idle or low load the fans would spin at 100%. Thermald was not doing it's job for whatever reason. It is hard to say why since most temperature sensors seem to be working fine and report acceptable temperatures.

I found a simple but great project called mbpfan which stopped the fans immediately after being started and still kept temperatures seemingly in check. I increased the minimum fan speed in mbpfan config just to avoid any potential overheating problems. With this setup I was getting 70-80 degrees with no overheating problems and a quiet fan.

CPU is in constant low frequency state (dealbreaker)

After installing cpufreq gnome extension I figured out that CPU is always at 800MHz. Mbpfan was not the cause since the same lack of scaling appeared when it was turned off.

First I tried to disable Intel p_state driver but the lack of scaling continued. Using userspace driver in cpufreq, I was unable to change min/max frequencies or force a specific frequency via cpupower.

As per ArchWiki, I gifured out that BIOS was enforcing this state via

cat /sys/devices/system/cpu/cpu0/cpufreq/bios_limit

After ignoring ppc via

echo 1 > /sys/module/processor/parameters/ignore_ppc

the CPU instantly started to scale as expected. Unfortunately this was not the final solution since the low state  would randomly reappear again for long periods of time with small time windows of scaling working as expected. Therefore, even with ignore_ppc I would still get 800MHz most of the time with temps reported around 70 degrees.

In this state Gnome Shell would lag and everything was half-usable.

Something in hardware was throttling CPU and I wasn't able to overcome it.

Bcmwl driver very spotty

WiFi bcmwl driver is very spotty. It would connect to Android hotspot no problem but it failed to connect to WiFi router. Small sample of 50% reliability.

Display positions not remembered after reboot

I had to rearrange the external monitors on each reboot since Gnome would not remember their positions. I had to come up with xrandr script to run after login to remedy this sad state of affairs.

No per-monitor scaling

Gnome still does not support setting the scale factors per monitor. Again, I had to come up with xrandr script to achieve 200% scaling on HiDPI and regular scaling on external monitors (1900×1200).

Broken scaling under Wayland

Apparently if you set the scale factor to default in Wayland session, things should "just work" across HiDPI and non-HiDPI displays. Don't believe these people, they are liars.

I set the scaling to default but that made HiDPI desktop tiny while external monitors were fine. Increasing the scaling to 200% made HiDPI fine while external monitors scaled also.

There is also no xrandr under Wayland so you can't help yourself with that.

Broken rendering of electron apps on external monitors

If using Postman on an external monitor, parts of the dialog boxes would simply disappear, making the tool unusable. Using Postman on main display did not have this issue. Weird.

 

The bottom line: get a Dell or a Lenovo for your Linux workstation needs.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Xrandr framebuffer and per-display scaling

Ubuntu 18.04 LTS came out recently with Gnome desktop as default. Unfortunately even in 2018, it won't remember the external monitor positioning after reboot and has no support in display settings to set per-display scaling. Year of the Linux desktop, anyone?

Xrandr is a powerful Linux tool to manipulate displays. Unfortunately, the man page is very sparse on information with badly explained flags and various Linux guides are no better.

This example will create a triple monitor setup with HiDPI laptop display at the bottom of the array.

Xrandr command:

xrandr --fbmm 11520x4200 --output eDP --pos 4320x2400 --mode 2880x1800 --scale 1x1 --primary --output DisplayPort-1 --pos 0x0 --mode 1920x1200 --scale 2x2 --output DisplayPort-0 --pos 3840x0 --mode 1920x1200 --scale 2x2 --output HDMI-0 --pos 7680x0 --mode 1920x1200 --scale 2x2

 

Gnome scaling is set to 200% so our HiDPI native display looks normal. Unfortunately this also means non-HiDPI displays have this scaling applied which is not what we want.

Framebuffer is the full outer rectangle which must be able to contain our display setup as a whole.

Since external displays are scaled 2×2 (zoom out), they take twice the size of their actual resolution in our framebuffer. Meaning their sizes in fb are actually 3840×2400.

Y of the framebuffer is therefore 2400+1800=4200 (HiDPI display is scaled 1×1 so it takes the same amount of space in the framebuffer as it's resolution).

X of the framebuffer is 3*3840=11520.

–fbmm specifies the full framebuffer size

–pos specifies the position of the display in the buffer. 0x0 position starts on top-left corner.

–mode sets the actual display resolution

–output specifies the display output (run xrandr to list all available)

–scale specifies "zooming" in (<1) or out (>1)

 

The end result has some invisible area on the bottom-left and bottom-right corners so it is not ideal. I have yet to figure out if it is possible to specify fencing around that area.

The HiDPI display is also not perfectly aligned with the top display but that could be corrected with fractional scaling. It didn't really bother me to fiddle with that.

Finally, you should run this command with a startup script so you get the correct monitor positioning and scaling after login.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Running multiple PHP versions and compiling from source

We had a peculiar situation where both PHP5 and PHP7 were needed at the same time on a FreeBSD server. It was also prohibitive to get root or sudo access on the managing account. The solution was to compile PHP from source with fpm, run fpm on a UNIX socket and wire the specific domain which needed PHP7 through Nginx.

Compiling from source

Compiling PHP is pretty straightforward but you can easily forget crucial configure flags. First get the source archive. After trial and error, this appears to be "good enough" for Laravel 5.6 requirements.

./configure -prefix=/home/foo/php7 --with-openssl --enable-mbstring --with-pdo-mysql --enable-fpm --with-config-file-path=/home/foo/php7 --with-config-file-scan-dir=/home/foo/php7 --with-iconv --with-gmp --with-gd

Followed by

make && make install

copy php.ini-development file from source to the –with-config-file-path directory and call it php.ini.  In PREFIX/bin confirm that php binary is loading your ini file by running

./php --ini

If you made a mistake in your configure run, you have to make clean first or the changed options won't be picked up.

Configuring fpm

In your PREFIX/etc folder, check php-fpm.conf. The only lines I bothered to verify and edit were pid, error_log and include, the rest can be left as is.

Inside folder php-fpm.d create a config file like my.subdomain.conf and add something like

listen = /location/of/php-fpm.sock
user = foo
group = foo
pm = dynamic
pm.max_children = 5
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 1
pm.max_requests = 200
listen.backlog = -1
pm.status_path = /status
request_terminate_timeout = 30s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes
env[HOSTNAME] = $HOSTNAME
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

You can study fpm config in great details but this is enough to get you going.

Now you can run your PHP7 fpm process in PREFIX/sbin with

./php-fpm --fpm-config /prefix/etc/php-fpm.conf

Since we don't have any init scripts in this case, we must be able to kill it also. Find it with

ps aux | grep fpm

and kill with

kill -QUIT pid

Make sure it says php-fpm: master process in the ps output, killing children doesn't do anything, just respawns them.

Now you just need to wire up your Nginx to use this fpm instead of the PHP5 system one by specifying

fastcgi_pass unix:/location/of/php-fpm.sock;

We now have PHP dualstack with some web apps running on system provided PHP5 and some on our own built PHP7. We did not need root for this process, except possibly for configuring Nginx.

One obvious downside is that you now have to take care of updating your PHP version on your own but that is one view we had to sacrifice. Updates should be relatively easy and fast since make install won't remove your config files in existing installation and compiling PHP takes just a few minutes.

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Centos Docker unable to unmount on stop/start

Error looks something like

Error response from daemon: driver "overlay" failed to remove root filesystem for 805c245dad451542b44bb1b58c60887fa98a64a61f2f0b8de32fa5b13ccc8ce4: remove /var/lib/docker/overlay/8f666b802f418f4a3dc4a6cafbefa79afc81491a5cb23da8084dd14e33afbea0/merged: device or resource busy

It usually appears when stopping and starting a container. It results in Docker container not being able to start or starts in a weird dead/removed state. Issue is presumably fixed in Centos 7.4 as per this github issue, but a workaround exists:

1. Grep by part of ID:

grep docker /proc/*/mountinfo | grep 8f66

it shows which process is holding up the mount in busy state

/proc/31415/mountinfo:…

2. Kill the process

sudo kill 31415

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Lib packaging for your own repo

This is a note to self about the release procedure and distro packaging of a development library.

This instructions expect aptly and createrepo to be preinstalled on your repo server together with a valid GPG key.

Packaging a .deb

1. Checkout the release tag and build the deb according to README. Each build should be done on the same machine as the target distribution. If I am packaging for Centos 6 I am also building on Centos 6. If CMake and CPack are set up correctly it usually boils down to:

cmake -G "Unix makefiles" -H./ -B./build
cd build
make
cpack -G "DEB"

If the project is missing CMake, refuse to package it.

2. Check that deb info is correct:

dpkg-deb -I bncsutil-1.4.1-Linux.deb

3. Rename it to distro you are building on, then scp to your repo server

mv bncsutil-1.4.1-Linux.deb bncsutil-1.4.1-Debian9.deb

4. If aptly repo does not exist yet, create it

aptly repo create -comment="Debian 9 repo" -component="main" -distribution="bnetdocs-stretch" debian9

Ideally you only create the repo the first time, for future updates you create a snapshot of it, add a package, then switch the repo to new snapshot. See aptly docs for more.

Alternatively, you can just add more packages and update the repo with

aptly publish update bnetdocs-stretch debian9

 

5. Add package to repo

aptly repo add debian9 deb_archives/bncsutil-1.4.1-Debian9.deb

6. Publish repo

aptly publish repo debian9 debian9

7. On target machine, add repo to /etc/apt/sources.list and fetch public key

deb http://apt.xpam.pl/debian8/ bnetdocs-stretch main
wget -qO - https://apt.xpam.pl/xpam.pl-pubkey.asc | sudo apt-key add -

8. Update and then check if package info is correct

apt-cache show bncsutil

9. If big mistakes were made

aptly publish drop debian9 debian9

..and start over. Repeat for Debian 8 etc.

Packaging an .rpm

1. See the previous #1. The only difference is

cpack -G "RPM"

2. Check that rpm info is correct

rpm -qip bncsutil-1.4.1-Linux.rpm

3. Rename it to distro you are building on, then scp to your repo server

mv bncsutil-1.4.1-Linux.rpm bncsutil-1.4.1-Centos7.rpm

4. Make sure you have .rpmmacros file in home dir with uid of gpg signing key (check out your keys with gpg –list-keys). If you don't have one, generate it. Entry in the file should look like:

%_gpg_name <uid here>

5. Sign rpm

rpm --addsign rpm_archives/bncsutil-1.4.1-Centos7.rpm

4. Move to appropriate repo that was created by createrepo earlier (see createrepo docs)

mv rpm_archives/bncsutil-1.4.1-Centos7.rpm .createrepo-centos7/

5. Update repo metadata

createrepo --update ./.createrepo-centos7/

6. Add your repo on the target machine

yum -y install yum-utils
yum-config-manager --add-repo https://centos7.rpm.xpam.pl
yum-config-manager --enable https://centos7.rpm.xpam.pl
rpm --import https://centos7.rpm.xpam.pl/xpam.pl-pubkey.asc

Since createrepo is pretty much just an http server you can simply delete an rpm and update the metadata in case things go south.

Repeat for Centos 6 etc.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs