Category Archives: Script magic

Setting env variables with hyphen and running a program

Docker compose allows you very unrestrictive naming of your environment variables. It allows you to use hyphen and other "special" characters in variables names. When you need to use these variables in regular shell you are out of luck, bash and many other shells do not allow hyphens in variable names. But this is merely a shell restriction, so how to do it?

With env

env -i 'TZ=Europe/Berlin' \
'PORT=8080' \
'BASE-URL=http://localhost:8080' \
'DB[0]_CONNECTION-URL=jdbc:postgresql://localhost:5432/postgres' \
'DB[0]_USERNAME=username' \
'DB[0]_PASSWORD=password' java -jar myapp.jar

Note that env ignores all inherited env variables so you might need to redefine them:

env -i JAVA_HOME=$JAVA_HOME \
'TZ=Europe/Berlin' \ 
'PORT=8080' java -jar myapp.jar

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Creating a new torrent and seeding with Transmission

You have setup a Transmission server on your Linux box together with Transmisison Web or something along those lines and now you are wondering.. how can I actually seed a NEW file?

I couldn't find a straightforward answer on the web so here is the short tutorial:

  1. Upload your file to your transmission download directory
  2. cd to that directory and create a torrent file (lets say the file you uploaded was called  myfile.rar):
    transmission-create -o myfile.torrent -c "this is my file comment" -t tracker1 -t tracker2 -t tracker3 myfile.rar

    Replace tracker1, tracker3, tracker3, …trackerN with a bunch of trackers. Better specify more than one in case they go down. Here is a cool little list of public trackers.

  3. Download the new .torrent file you just created, open Transmission Web and add the torrent. Since the file already exists in download directory, Transmisison will just revalidate the data and start seeding. *mind blown*
  4. Distribute the torrent file to your people or generate a magnet link with
    transmission-show -m myfile.torrent

     

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Xrandr framebuffer and per-display scaling

Ubuntu 18.04 LTS came out recently with Gnome desktop as default. Unfortunately even in 2018, it won't remember the external monitor positioning after reboot and has no support in display settings to set per-display scaling. Year of the Linux desktop, anyone?

Xrandr is a powerful Linux tool to manipulate displays. Unfortunately, the man page is very sparse on information with badly explained flags and various Linux guides are no better.

This example will create a triple monitor setup with HiDPI laptop display at the bottom of the array.

Xrandr command:

xrandr --fbmm 11520x4200 --output eDP --pos 4320x2400 --mode 2880x1800 --scale 1x1 --primary --output DisplayPort-1 --pos 0x0 --mode 1920x1200 --scale 2x2 --output DisplayPort-0 --pos 3840x0 --mode 1920x1200 --scale 2x2 --output HDMI-0 --pos 7680x0 --mode 1920x1200 --scale 2x2

 

Gnome scaling is set to 200% so our HiDPI native display looks normal. Unfortunately this also means non-HiDPI displays have this scaling applied which is not what we want.

Framebuffer is the full outer rectangle which must be able to contain our display setup as a whole.

Since external displays are scaled 2×2 (zoom out), they take twice the size of their actual resolution in our framebuffer. Meaning their sizes in fb are actually 3840×2400.

Y of the framebuffer is therefore 2400+1800=4200 (HiDPI display is scaled 1×1 so it takes the same amount of space in the framebuffer as it's resolution).

X of the framebuffer is 3*3840=11520.

–fbmm specifies the full framebuffer size

–pos specifies the position of the display in the buffer. 0x0 position starts on top-left corner.

–mode sets the actual display resolution

–output specifies the display output (run xrandr to list all available)

–scale specifies "zooming" in (<1) or out (>1)

 

The end result has some invisible area on the bottom-left and bottom-right corners so it is not ideal. I have yet to figure out if it is possible to specify fencing around that area.

The HiDPI display is also not perfectly aligned with the top display but that could be corrected with fractional scaling. It didn't really bother me to fiddle with that.

Finally, you should run this command with a startup script so you get the correct monitor positioning and scaling after login.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Running multiple PHP versions and compiling from source

We had a peculiar situation where both PHP5 and PHP7 were needed at the same time on a FreeBSD server. It was also prohibitive to get root or sudo access on the managing account. The solution was to compile PHP from source with fpm, run fpm on a UNIX socket and wire the specific domain which needed PHP7 through Nginx.

Compiling from source

Compiling PHP is pretty straightforward but you can easily forget crucial configure flags. First get the source archive. After trial and error, this appears to be "good enough" for Laravel 5.6 requirements.

./configure -prefix=/home/foo/php7 --with-openssl --enable-mbstring --with-pdo-mysql --enable-fpm --with-config-file-path=/home/foo/php7 --with-config-file-scan-dir=/home/foo/php7 --with-iconv --with-gmp --with-gd

Followed by

make && make install

copy php.ini-development file from source to the –with-config-file-path directory and call it php.ini.  In PREFIX/bin confirm that php binary is loading your ini file by running

./php --ini

If you made a mistake in your configure run, you have to make clean first or the changed options won't be picked up.

Configuring fpm

In your PREFIX/etc folder, check php-fpm.conf. The only lines I bothered to verify and edit were pid, error_log and include, the rest can be left as is.

Inside folder php-fpm.d create a config file like my.subdomain.conf and add something like

listen = /location/of/php-fpm.sock
user = foo
group = foo
pm = dynamic
pm.max_children = 5
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 1
pm.max_requests = 200
listen.backlog = -1
pm.status_path = /status
request_terminate_timeout = 30s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes
env[HOSTNAME] = $HOSTNAME
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

You can study fpm config in great details but this is enough to get you going.

Now you can run your PHP7 fpm process in PREFIX/sbin with

./php-fpm --fpm-config /prefix/etc/php-fpm.conf

Since we don't have any init scripts in this case, we must be able to kill it also. Find it with

ps aux | grep fpm

and kill with

kill -QUIT pid

Make sure it says php-fpm: master process in the ps output, killing children doesn't do anything, just respawns them.

Now you just need to wire up your Nginx to use this fpm instead of the PHP5 system one by specifying

fastcgi_pass unix:/location/of/php-fpm.sock;

We now have PHP dualstack with some web apps running on system provided PHP5 and some on our own built PHP7. We did not need root for this process, except possibly for configuring Nginx.

One obvious downside is that you now have to take care of updating your PHP version on your own but that is one view we had to sacrifice. Updates should be relatively easy and fast since make install won't remove your config files in existing installation and compiling PHP takes just a few minutes.

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Centos Docker unable to unmount on stop/start

Error looks something like

Error response from daemon: driver "overlay" failed to remove root filesystem for 805c245dad451542b44bb1b58c60887fa98a64a61f2f0b8de32fa5b13ccc8ce4: remove /var/lib/docker/overlay/8f666b802f418f4a3dc4a6cafbefa79afc81491a5cb23da8084dd14e33afbea0/merged: device or resource busy

It usually appears when stopping and starting a container. It results in Docker container not being able to start or starts in a weird dead/removed state. Issue is presumably fixed in Centos 7.4 as per this github issue, but a workaround exists:

1. Grep by part of ID:

grep docker /proc/*/mountinfo | grep 8f66

it shows which process is holding up the mount in busy state

/proc/31415/mountinfo:…

2. Kill the process

sudo kill 31415

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Lib packaging for your own repo

This is a note to self about the release procedure and distro packaging of a development library.

This instructions expect aptly and createrepo to be preinstalled on your repo server together with a valid GPG key.

Packaging a .deb

1. Checkout the release tag and build the deb according to README. Each build should be done on the same machine as the target distribution. If I am packaging for Centos 6 I am also building on Centos 6. If CMake and CPack are set up correctly it usually boils down to:

cmake -G "Unix makefiles" -H./ -B./build
cd build
make
cpack -G "DEB"

If the project is missing CMake, refuse to package it.

2. Check that deb info is correct:

dpkg-deb -I bncsutil-1.4.1-Linux.deb

3. Rename it to distro you are building on, then scp to your repo server

mv bncsutil-1.4.1-Linux.deb bncsutil-1.4.1-Debian9.deb

4. If aptly repo does not exist yet, create it

aptly repo create -comment="Debian 9 repo" -component="main" -distribution="bnetdocs-stretch" debian9

Ideally you only create the repo the first time, for future updates you create a snapshot of it, add a package, then switch the repo to new snapshot. See aptly docs for more.

Alternatively, you can just add more packages and update the repo with

aptly publish update bnetdocs-stretch debian9

 

5. Add package to repo

aptly repo add debian9 deb_archives/bncsutil-1.4.1-Debian9.deb

6. Publish repo

aptly publish repo debian9 debian9

7. On target machine, add repo to /etc/apt/sources.list and fetch public key

deb http://apt.xpam.pl/debian8/ bnetdocs-stretch main
wget -qO - https://apt.xpam.pl/xpam.pl-pubkey.asc | sudo apt-key add -

8. Update and then check if package info is correct

apt-cache show bncsutil

9. If big mistakes were made

aptly publish drop debian9 debian9

..and start over. Repeat for Debian 8 etc.

Packaging an .rpm

1. See the previous #1. The only difference is

cpack -G "RPM"

2. Check that rpm info is correct

rpm -qip bncsutil-1.4.1-Linux.rpm

3. Rename it to distro you are building on, then scp to your repo server

mv bncsutil-1.4.1-Linux.rpm bncsutil-1.4.1-Centos7.rpm

4. Make sure you have .rpmmacros file in home dir with uid of gpg signing key (check out your keys with gpg –list-keys). If you don't have one, generate it. Entry in the file should look like:

%_gpg_name <uid here>

5. Sign rpm

rpm --addsign rpm_archives/bncsutil-1.4.1-Centos7.rpm

4. Move to appropriate repo that was created by createrepo earlier (see createrepo docs)

mv rpm_archives/bncsutil-1.4.1-Centos7.rpm .createrepo-centos7/

5. Update repo metadata

createrepo --update ./.createrepo-centos7/

6. Add your repo on the target machine

yum -y install yum-utils
yum-config-manager --add-repo https://centos7.rpm.xpam.pl
yum-config-manager --enable https://centos7.rpm.xpam.pl
rpm --import https://centos7.rpm.xpam.pl/xpam.pl-pubkey.asc

Since createrepo is pretty much just an http server you can simply delete an rpm and update the metadata in case things go south.

Repeat for Centos 6 etc.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Bashmagic collection vol1

Keep only last X lines of a file (shrink).

echo "$(tail -n 10000 huge.log)" > huge.log

Deploy maven artefact to a specific repo without specifying it in pom.xml (format repoId:default:repoUrl). Repo should be specified in your .m2 settings.xml with any necessary credentials.

mvn deploy -DaltDeploymentRepository=repo.mydomain.com.my.releases::default::https://repo.mydomain.com/repository/maven-my-releases/

One liner to set password for default PostgreSQL user after initial install.

runuser -l postgres -c $'psql -c "ALTER USER postgres WITH PASSWORD \'postgres\';"'

Run command inside a screen and save all output to a file

screen -dm bash -c 'script -c "some chatty command" output.txt'

Scroll around inside a screen.

screen -r myscreen

ctrl+a

[

ctrl+u and ctrl+d for up and down. bufer is limited

Add 4GB swap on Centos 7 with a stroke of a copy-paste.

sudo dd if=/dev/zero of=/swapfile count=4096 bs=1MiB
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Standard tcpdump.

tcpdump -i eth0 -s 65535 -w dump.pcap

Show listening ports with corresponding executables.

netstat -tulpn

Show systemd logs for a specific service.

journalctl -u nginx.service

Freshly installed nginx configured as reverse proxy on Centos 7 getting "Permission denied" when connecting to backend service

setsebool -P httpd_can_network_connect 1

Convert a certificate stored in a Java keystore to a PEM cert and key (for example, Tomcat to Nginx transition).

$JAVA_HOME/bin/keytool -importkeystore -srckeystore .keystore -destkeystore keystore.p12 -deststoretype PKCS12 -srcalias mycrtalias -deststorepass changeit -destkeypass changeit
openssl pkcs12 -in keystore.p12 -nokeys -out cert.pem
openssl pkcs12 -in keystore.p12 -nodes -nocerts -out key.pem

Disable git SSL verification per-repo.

git config http.sslVerify "false"

Disable git SSL verification at clone time.

git -c http.sslVerify=false clone something

Clear git username and password cache for a repo (in case of password change or similar).

git config --global --unset user.password

Give user sudo privileges.

sudo usermod -aG wheel <user>

Git submodule is added to an existing repo and is not resolving for you locally.

git submodule update --init --recursive

Print all TCP connections of a Docker container.

sudo docker inspect -f '{{.State.Pid}}' containerName
<result>
sudo nsenter -t <result> -n netstat

Nmap portscan.

sudo nmap -p 1-65535 -sV -sS -T4 <ip>

Force JVM to use /dev/urandom instead of /dev/random (sometimes needed in low entropy environments like Docker).

java -Djava.security.egd=file:/dev/./urandom ...

Debug print all network activities on JVM level.

java -Djavax.net.debug=all ...

Create .htpasswd file for Nginx

sudo sh -c "echo -n 'someuser:' >> /etc/nginx/.htpasswd"
sudo sh -c "openssl passwd -apr1 >> /etc/nginx/.htpasswd"

Do or do not do something if file was modified recently

RECENTLY_CHANGED=$(find /tmp/me -newermt '10 minutes ago' |wc -l |xargs)
if [ "$RECENTLY_CHANGED" -eq 1 ]; then
    echo "File was changed recently, terminating script"
    exit 0
fi

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Periodically pause and resume Transmission torrents on FreeNAS

Unless you have great internet bandwidth you probably don't want to leech and seed your 100 torrents during the day. You usually want them to seed during the night and then pause in the morning. Let's do this on our FreeNAS box.

  1.  Open up your jails, select transmission and fire up the shell from the icon below.
  2. If you dislike vi as default editor, set it to ee by running the following command:
    setenv EDITOR /usr/bin/ee

    To make this choice persistent, open up .cshrc and add the command on the bottom.

  3. Run
    crontab -e

    and write the following lines:

    0 1 * * * /usr/local/bin/transmission-remote localhost:9091 --auth=YOUR_TRANSMISSION_USER:YOUR_TRANSMISSION_PASSWORD -tall --start
    0 7 * * * /usr/local/bin/transmission-remote localhost:9091 --auth=YOUR_TRANSMISSION_USER:YOUR_TRANSMISSION_PASSWORD -tall --stop

    Change the username and password with the ones you use to access Transmission WebUI. The example will start all torrents at 1AM and stop them at 7AM.

 

Update in 2023: in TrueNAS 13 (perhaps even earlier versions), transmission-remote is no longer preinstalled inside the jail, you need to install it first:

pkg install transmission-utils

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs