Category Archives: Linux

Centos Docker unable to unmount on stop/start

Error looks something like

Error response from daemon: driver "overlay" failed to remove root filesystem for 805c245dad451542b44bb1b58c60887fa98a64a61f2f0b8de32fa5b13ccc8ce4: remove /var/lib/docker/overlay/8f666b802f418f4a3dc4a6cafbefa79afc81491a5cb23da8084dd14e33afbea0/merged: device or resource busy

It usually appears when stopping and starting a container. It results in Docker container not being able to start or starts in a weird dead/removed state. Issue is presumably fixed in Centos 7.4 as per this github issue, but a workaround exists:

1. Grep by part of ID:

grep docker /proc/*/mountinfo | grep 8f66

it shows which process is holding up the mount in busy state

/proc/31415/mountinfo:…

2. Kill the process

sudo kill 31415

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Lib packaging for your own repo

This is a note to self about the release procedure and distro packaging of a development library.

This instructions expect aptly and createrepo to be preinstalled on your repo server together with a valid GPG key.

Packaging a .deb

1. Checkout the release tag and build the deb according to README. Each build should be done on the same machine as the target distribution. If I am packaging for Centos 6 I am also building on Centos 6. If CMake and CPack are set up correctly it usually boils down to:

cmake -G "Unix makefiles" -H./ -B./build
cd build
make
cpack -G "DEB"

If the project is missing CMake, refuse to package it.

2. Check that deb info is correct:

dpkg-deb -I bncsutil-1.4.1-Linux.deb

3. Rename it to distro you are building on, then scp to your repo server

mv bncsutil-1.4.1-Linux.deb bncsutil-1.4.1-Debian9.deb

4. If aptly repo does not exist yet, create it

aptly repo create -comment="Debian 9 repo" -component="main" -distribution="bnetdocs-stretch" debian9

Ideally you only create the repo the first time, for future updates you create a snapshot of it, add a package, then switch the repo to new snapshot. See aptly docs for more.

Alternatively, you can just add more packages and update the repo with

aptly publish update bnetdocs-stretch debian9

 

5. Add package to repo

aptly repo add debian9 deb_archives/bncsutil-1.4.1-Debian9.deb

6. Publish repo

aptly publish repo debian9 debian9

7. On target machine, add repo to /etc/apt/sources.list and fetch public key

deb http://apt.xpam.pl/debian8/ bnetdocs-stretch main
wget -qO - https://apt.xpam.pl/xpam.pl-pubkey.asc | sudo apt-key add -

8. Update and then check if package info is correct

apt-cache show bncsutil

9. If big mistakes were made

aptly publish drop debian9 debian9

..and start over. Repeat for Debian 8 etc.

Packaging an .rpm

1. See the previous #1. The only difference is

cpack -G "RPM"

2. Check that rpm info is correct

rpm -qip bncsutil-1.4.1-Linux.rpm

3. Rename it to distro you are building on, then scp to your repo server

mv bncsutil-1.4.1-Linux.rpm bncsutil-1.4.1-Centos7.rpm

4. Make sure you have .rpmmacros file in home dir with uid of gpg signing key (check out your keys with gpg –list-keys). If you don't have one, generate it. Entry in the file should look like:

%_gpg_name <uid here>

5. Sign rpm

rpm --addsign rpm_archives/bncsutil-1.4.1-Centos7.rpm

4. Move to appropriate repo that was created by createrepo earlier (see createrepo docs)

mv rpm_archives/bncsutil-1.4.1-Centos7.rpm .createrepo-centos7/

5. Update repo metadata

createrepo --update ./.createrepo-centos7/

6. Add your repo on the target machine

yum -y install yum-utils
yum-config-manager --add-repo https://centos7.rpm.xpam.pl
yum-config-manager --enable https://centos7.rpm.xpam.pl
rpm --import https://centos7.rpm.xpam.pl/xpam.pl-pubkey.asc

Since createrepo is pretty much just an http server you can simply delete an rpm and update the metadata in case things go south.

Repeat for Centos 6 etc.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Bashmagic collection vol1

Keep only last X lines of a file (shrink).

echo "$(tail -n 10000 huge.log)" > huge.log

Deploy maven artefact to a specific repo without specifying it in pom.xml (format repoId:default:repoUrl). Repo should be specified in your .m2 settings.xml with any necessary credentials.

mvn deploy -DaltDeploymentRepository=repo.mydomain.com.my.releases::default::https://repo.mydomain.com/repository/maven-my-releases/

One liner to set password for default PostgreSQL user after initial install.

runuser -l postgres -c $'psql -c "ALTER USER postgres WITH PASSWORD \'postgres\';"'

Run command inside a screen and save all output to a file

screen -dm bash -c 'script -c "some chatty command" output.txt'

Scroll around inside a screen.

screen -r myscreen

ctrl+a

[

ctrl+u and ctrl+d for up and down. bufer is limited

Add 4GB swap on Centos 7 with a stroke of a copy-paste.

sudo dd if=/dev/zero of=/swapfile count=4096 bs=1MiB
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Standard tcpdump.

tcpdump -i eth0 -s 65535 -w dump.pcap

Show listening ports with corresponding executables.

netstat -tulpn

Show systemd logs for a specific service.

journalctl -u nginx.service

Freshly installed nginx configured as reverse proxy on Centos 7 getting "Permission denied" when connecting to backend service

setsebool -P httpd_can_network_connect 1

Convert a certificate stored in a Java keystore to a PEM cert and key (for example, Tomcat to Nginx transition).

$JAVA_HOME/bin/keytool -importkeystore -srckeystore .keystore -destkeystore keystore.p12 -deststoretype PKCS12 -srcalias mycrtalias -deststorepass changeit -destkeypass changeit
openssl pkcs12 -in keystore.p12 -nokeys -out cert.pem
openssl pkcs12 -in keystore.p12 -nodes -nocerts -out key.pem

Disable git SSL verification per-repo.

git config http.sslVerify "false"

Disable git SSL verification at clone time.

git -c http.sslVerify=false clone something

Clear git username and password cache for a repo (in case of password change or similar).

git config --global --unset user.password

Give user sudo privileges.

sudo usermod -aG wheel <user>

Git submodule is added to an existing repo and is not resolving for you locally.

git submodule update --init --recursive

Print all TCP connections of a Docker container.

sudo docker inspect -f '{{.State.Pid}}' containerName
<result>
sudo nsenter -t <result> -n netstat

Nmap portscan.

sudo nmap -p 1-65535 -sV -sS -T4 <ip>

Force JVM to use /dev/urandom instead of /dev/random (sometimes needed in low entropy environments like Docker).

java -Djava.security.egd=file:/dev/./urandom ...

Debug print all network activities on JVM level.

java -Djavax.net.debug=all ...

Create .htpasswd file for Nginx

sudo sh -c "echo -n 'someuser:' >> /etc/nginx/.htpasswd"
sudo sh -c "openssl passwd -apr1 >> /etc/nginx/.htpasswd"

Do or do not do something if file was modified recently

RECENTLY_CHANGED=$(find /tmp/me -newermt '10 minutes ago' |wc -l |xargs)
if [ "$RECENTLY_CHANGED" -eq 1 ]; then
    echo "File was changed recently, terminating script"
    exit 0
fi

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Oh FileZilla…

I have encountered a weird problem when connecting to our FreeBSD server with FileZilla over SFTP. Either with password or key authentication I would get:

Error:    Server sent disconnect message
Error:    type 2 (protocol error):
Error:    "Too many authentication failures"

So let's turn on debugging shall we?

 

Trace:    Pageant is running. Requesting keys.
Trace:    Pageant has 15 SSH-2 keys
Trace:    Successfully loaded 1 key pair from file
Trace:    Trying Pageant key #0
Trace:    Server refused our key
Trace:    Trying Pageant key #1
Trace:    Server refused our key
Trace:    Trying Pageant key #2
Trace:    Server refused our key
Trace:    Trying Pageant key #3
Trace:    Server refused our key
Trace:    Trying Pageant key #4
Trace:    Server refused our key
Trace:    Trying Pageant key #5
Trace:    Received disconnect message (protocol error)
Trace:    Disconnection message text: Too many authentication failures

So basically, I give Filezilla a specific keyfile but it tries all my keys anyway. Now let's see what the bright minds on FileZilla issue tracker have to say about this bug.

https://trac.filezilla-project.org/ticket/7739 gives us a workaround:

SSH_AUTH_SOCK=""; filezilla

which works nicely. A working workaround is a blessing if you really need to use someting that is essentially broken. The bug is marked as a duplicate of https://trac.filezilla-project.org/ticket/5480

This bug contains a brilliant comment by an apparent FileZilla developer:

This is by design, FileZilla uses the system's SSH agent.

Just reconfigure the server to allow for more keys.

What the actual? The bug will apparently be solved via https://trac.filezilla-project.org/ticket/8232

which is marked as "fixed" and the comment 19 months ago says it will be in the "next version". The latest version is 3.24.0 released on January 1st 2017 which is exactly what I have and guess what? Not fixed, after 7 years.

 

So at this point I'll just safely assume that FileZilla might as well be the worst SFTP client in existence and just use something else. But guess what? There is more. The exact same problem exists in Gnome Files if you try to open an sftp:// location. The obvious reason is that Gnome Files does not ask you anything about keys or athenticaton type but just cycles through SSH keys to try and find the correct one. Why did nobody think about offering me a popup dialog to pick the correct key? Probably because Gnome likes to dumb down things, I can't really find any other reason.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

pgadmin4 on Fedora 25

wget https://download.postgresql.org/pub/repos/yum/9.6/fedora/fedora-25-x86_64/pgdg-fedora96-9.6-3.noarch.rpm
sudo dnf install pgdg-fedora96-9.6-3.noarch.rpm
sudo dnf install pgadmin4-v1
sudo systemctl start pgadmin4-v1

You can now access the web interface at http://localhost:5050.

Unfortunately the standalone app does not currently work due to a bug in pgadmin4 package.

Fortunately the pgadmin4 standalone app is just a web wrapper so you are not missing much.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Fedora 25 on Lenoyo Y50

Everything except WiFi worked out of the box. To get the WiFi working:

sudo dnf install http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-25.noarch.rpm
sudo dnf install http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-25.noarch.rpm

# The rest
sudo dnf install kernel-devel broadcom-wl akmod-wl akmods
sudo akmods
sudo reboot

Taken from here.

In your BIOS make sure you disable Secure Boot. Not UEFI, not Legacy mode, specifically the switch that disables secure boot and nothing else. After these steps, WiFi works. To enter BIOS on Y50, tap F2 after Lenovo splash screen.

 

Edit 7.2.2017: WiFi performance is unfortunately ABYSMAL. Will update this post if I find any solutions. Connection is super slow and constantly dropping.

Luckily, USB tethering from Android works like a charm so it's not a deal breaker for now.

Edit 2: looks like blacklisting bcma driver makes things much much better:

sudo nano /etc/modprobe.d/blacklist.conf

#add this
blacklist bcma

sudo systemctl restart NetworkManager

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

OJDBC7 in a Docker container? Prepare for trouble

Scenario: A JDK8 Docker container using OJDBC7 to connect to the database. Sounds simple enough, what could go wrong?

Simptoms: Connecting to the database randomly takes several minutes, fails with a weird SqlRecoverableException: no more data to read from socket or just works fine as if there is no problem.

The same Docker image also works fine on some machine but fails consistently on other.

The reason is this. Docker is not good at /dev/random. Probably even more so if you run it in a VM, since it's double isolated from actual entropy sources (my non scientific observation). For whatever reason, OJDBC defaults to /dev/random and this causes a block when connecting to the database due to high probability of /dev/random depletion.

Simple solution is to just mount /dev/urandom to /dev/random inside the Docker, in docker run command:

-v /dev/urandom:/dev/random

So.. if you ever want to use OJDBC inside Docker, remember this flag. It will save lives or at least spare you hours of useless debugging.

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Fedora 24 XFCE pains and gains

I recently installed Fedora 24 XFCE on my brand new Entroware Apollo, a Linux friendly laptop. Here is a list of problems I encountered during my first week of use. While most of these problems had a solution with a bit of googling, non-tech savvy person would have severe problems solving them.

PAINS

  1. System completely hangs when I connect a second monitor. Unresolved.
  2. Could not see any WiFi networks because the interface was not managed. Had to make it managed in NetworkManager.conf manually.
  3. At some point, XFCE panel would not show up anymore and I got an error message on startup asking me to start the panel. Solved it by deleting .config folder. What the hell?
  4. No login prompt when laptop comes out of suspend, even though I have both option in power management and session settings turned on. Unresolved.
  5. Adding programs to favorites in Whiskers menu sometimes does not persist across reboots. I think esepcially when I did a hard reset due to issue #1.
  6. My Nexus 4 would not automount in Thunar over USB. After an hour of Googling and installing random packages I got it to work, I think?

GAINS

  1. Numix theme on XFCE is extremely nice, I love it. There is just one small bug so far in Volume widget, the selection color hides the slider completely.
  2. My FreeNAS was automatically detected by Thunar while the old Gnome-files setup would not show it and even refuse to mount it sometimes manually. A nice surprise.
  3. I like how you can fully customize XFCE panels. The only thing I actually couldn't do is completely hide the panel (there is like a 3px grey bar when the panel is hidden). Other than that it's super nice.
  4. Fedy is super nice. Props to it's maintainers, it all just works.

 

So basically, everything kinda works now but it was a bumpy road to achieve this. Hopefully I get some feedback on issue #1 in Fedora bug tracker because it's really severe.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs