All posts by cen

Bashmagic collection vol1

Keep only last X lines of a file (shrink).

echo "$(tail -n 10000 huge.log)" > huge.log

Deploy maven artefact to a specific repo without specifying it in pom.xml (format repoId:default:repoUrl). Repo should be specified in your .m2 settings.xml with any necessary credentials.

mvn deploy -DaltDeploymentRepository=repo.mydomain.com.my.releases::default::https://repo.mydomain.com/repository/maven-my-releases/

One liner to set password for default PostgreSQL user after initial install.

runuser -l postgres -c $'psql -c "ALTER USER postgres WITH PASSWORD \'postgres\';"'

Run command inside a screen and save all output to a file

screen -dm bash -c 'script -c "some chatty command" output.txt'

Scroll around inside a screen.

screen -r myscreen

ctrl+a

[

ctrl+u and ctrl+d for up and down. bufer is limited

Add 4GB swap on Centos 7 with a stroke of a copy-paste.

sudo dd if=/dev/zero of=/swapfile count=4096 bs=1MiB
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Standard tcpdump.

tcpdump -i eth0 -s 65535 -w dump.pcap

Show listening ports with corresponding executables.

netstat -tulpn

Show systemd logs for a specific service.

journalctl -u nginx.service

Freshly installed nginx configured as reverse proxy on Centos 7 getting "Permission denied" when connecting to backend service

setsebool -P httpd_can_network_connect 1

Convert a certificate stored in a Java keystore to a PEM cert and key (for example, Tomcat to Nginx transition).

$JAVA_HOME/bin/keytool -importkeystore -srckeystore .keystore -destkeystore keystore.p12 -deststoretype PKCS12 -srcalias mycrtalias -deststorepass changeit -destkeypass changeit
openssl pkcs12 -in keystore.p12 -nokeys -out cert.pem
openssl pkcs12 -in keystore.p12 -nodes -nocerts -out key.pem

Disable git SSL verification per-repo.

git config http.sslVerify "false"

Disable git SSL verification at clone time.

git -c http.sslVerify=false clone something

Clear git username and password cache for a repo (in case of password change or similar).

git config --global --unset user.password

Give user sudo privileges.

sudo usermod -aG wheel <user>

Git submodule is added to an existing repo and is not resolving for you locally.

git submodule update --init --recursive

Print all TCP connections of a Docker container.

sudo docker inspect -f '{{.State.Pid}}' containerName
<result>
sudo nsenter -t <result> -n netstat

Nmap portscan.

sudo nmap -p 1-65535 -sV -sS -T4 <ip>

Force JVM to use /dev/urandom instead of /dev/random (sometimes needed in low entropy environments like Docker).

java -Djava.security.egd=file:/dev/./urandom ...

Debug print all network activities on JVM level.

java -Djavax.net.debug=all ...

Create .htpasswd file for Nginx

sudo sh -c "echo -n 'someuser:' >> /etc/nginx/.htpasswd"
sudo sh -c "openssl passwd -apr1 >> /etc/nginx/.htpasswd"

Do or do not do something if file was modified recently

RECENTLY_CHANGED=$(find /tmp/me -newermt '10 minutes ago' |wc -l |xargs)
if [ "$RECENTLY_CHANGED" -eq 1 ]; then
    echo "File was changed recently, terminating script"
    exit 0
fi

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Watching beIN SPORTS Direct Spain from anywhere in the world

Update 2023: this is a very old blog post now and most probably does not work anymore

It's 2017 and it is almost impossible for a regular human being to watch Champions League without being subscribed to cable or internet TV and pay exorbitant fees for hundreds of bundled channels you don't really need or want.

beIN SPORTS Direct is 9.99€ a month in Spain which is a reasonable price. The problem is that it is geoblocked to Spain only. Here is a guide on how to overcome this.

  1. Register account

    Go to https://tiendas.beinconnect.es/contratar/promo/bein and insert your email and password. This will open up the following form:

    Nombre: your name
    Apellidos: surname
    Tipo documento: NIF
    NIF number: NIF is some kind of tax ID in Spain. It consists of 8 digits plus CRC character on the end. beIN sports has a validator on this number so it must be correct. Go to http://www.cespedes.org/dni2nif/ and insert random 8 digits, then copy the digits and the CRC.
    Teléfono móvil: This has to be a valid Spanish phone number. it starts with a 6 or 7 followed by 8 digits. You can put whatever valid number you like, there is no SMS confirmation or anything like that.
    Código postal: valid Spanish postal code. Pick whatever: https://en.wikipedia.org/wiki/List_of_postal_codes_in_Spain

    Finish up by giving your credit card info.

  2. VPN

    You should now have an account but if you try to play any of the streams they will be black unless you already have a Spanish IP. All you need to do now is find a decently priced VPN that allows you to freely switch locations and has servers in Spain. For a quick test you can do it via TunneloVPN Chrome xtension https://chrome.google.com/webstore/detail/tunnello-vpn-unblock-ultr/hoapmlpnmpaehilehggglehfdlnoegck which offers 1GB for free upon registration. In the extension choose Spain from the drop-down, turn it on and refresh your beIN SPORTS Direct page. Video should now start playing. You can verify that you have Spanish IP by going to any of the "what is my ip" websites.

And that's it, you can now watch beIN SPORTS Direct for 9.99€ a month from any country. You'll probably have to add 5€ to that monthly bill for a VPN but such is life. This is the cheapest way for a cord cutter to watch Champions League and La Liga online, at least known to me.

The only bad news is no HD streams which sucks.. but we'll get to this in a few years hopefully.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Oh FileZilla…

I have encountered a weird problem when connecting to our FreeBSD server with FileZilla over SFTP. Either with password or key authentication I would get:

Error:    Server sent disconnect message
Error:    type 2 (protocol error):
Error:    "Too many authentication failures"

So let's turn on debugging shall we?

 

Trace:    Pageant is running. Requesting keys.
Trace:    Pageant has 15 SSH-2 keys
Trace:    Successfully loaded 1 key pair from file
Trace:    Trying Pageant key #0
Trace:    Server refused our key
Trace:    Trying Pageant key #1
Trace:    Server refused our key
Trace:    Trying Pageant key #2
Trace:    Server refused our key
Trace:    Trying Pageant key #3
Trace:    Server refused our key
Trace:    Trying Pageant key #4
Trace:    Server refused our key
Trace:    Trying Pageant key #5
Trace:    Received disconnect message (protocol error)
Trace:    Disconnection message text: Too many authentication failures

So basically, I give Filezilla a specific keyfile but it tries all my keys anyway. Now let's see what the bright minds on FileZilla issue tracker have to say about this bug.

https://trac.filezilla-project.org/ticket/7739 gives us a workaround:

SSH_AUTH_SOCK=""; filezilla

which works nicely. A working workaround is a blessing if you really need to use someting that is essentially broken. The bug is marked as a duplicate of https://trac.filezilla-project.org/ticket/5480

This bug contains a brilliant comment by an apparent FileZilla developer:

This is by design, FileZilla uses the system's SSH agent.

Just reconfigure the server to allow for more keys.

What the actual? The bug will apparently be solved via https://trac.filezilla-project.org/ticket/8232

which is marked as "fixed" and the comment 19 months ago says it will be in the "next version". The latest version is 3.24.0 released on January 1st 2017 which is exactly what I have and guess what? Not fixed, after 7 years.

 

So at this point I'll just safely assume that FileZilla might as well be the worst SFTP client in existence and just use something else. But guess what? There is more. The exact same problem exists in Gnome Files if you try to open an sftp:// location. The obvious reason is that Gnome Files does not ask you anything about keys or athenticaton type but just cycles through SSH keys to try and find the correct one. Why did nobody think about offering me a popup dialog to pick the correct key? Probably because Gnome likes to dumb down things, I can't really find any other reason.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

pgadmin4 on Fedora 25

wget https://download.postgresql.org/pub/repos/yum/9.6/fedora/fedora-25-x86_64/pgdg-fedora96-9.6-3.noarch.rpm
sudo dnf install pgdg-fedora96-9.6-3.noarch.rpm
sudo dnf install pgadmin4-v1
sudo systemctl start pgadmin4-v1

You can now access the web interface at http://localhost:5050.

Unfortunately the standalone app does not currently work due to a bug in pgadmin4 package.

Fortunately the pgadmin4 standalone app is just a web wrapper so you are not missing much.

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Keycloak OAuth endpoints for Postman/HTTP Clients

When testing REST services secured by Keycloak you need to retrieve access tokens via Postman or similar REST client. If you want to implement your own client that has to authenticate with a token you also need to know the Keycloak OpenID endpoints in order to retrieve the access token, refresh it or to end the session (logout).

Retreiving the tokens for a public client using username and password

Public client is typically used for web applications and other client side apps.

Method: POST
URL: https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token
Body type: x-www-form-urlencoded
Form fields:
client_id <my-client-name>
grant_type password
username <username>
password <password>

Retreiving the tokens for a confidential client using client secret

Confidential client is typically used for secure apps on the back-end.

Method: POST
URL: https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token
Body type: x-www-form-urlencoded
Form fields:
client_id <my-confidential-client-name>
grant_type client_credentials
client_secret <my-client-secret>

Retreive an access token with a refresh token

The first two methods will yield you an access token which you use in the Authorization HTTP header and a refresh token which you save for later. Refresh tokens have much longer expire time as access tokens. The idea is that when the access token expires you use the refresh token to get a new access token. This request also gives you a new refresh token so you can keep the session alive until maximum refresh token expire time is reached. Refresh token expire time equals the session expire time.

Method: POST
URL: https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token
Body type: x-www-form-urlencoded
Form fields:
client_id <my-client-name>
grant_type refresh_token
refresh_token <my-refresh-token>

Logout the session

To logout and invalidate the session, call a /logout endpoint with your refresh token. The validity of the refresh token is essentially the validity of your entire session.

Method: POST
URL: https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/logout
Body type: x-www-form-urlencoded
Form fields:
client_id <my-client-name>
refresh_token <my-refresh-token>
Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Fedora 25 on Lenoyo Y50

Everything except WiFi worked out of the box. To get the WiFi working:

sudo dnf install http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-25.noarch.rpm
sudo dnf install http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-25.noarch.rpm

# The rest
sudo dnf install kernel-devel broadcom-wl akmod-wl akmods
sudo akmods
sudo reboot

Taken from here.

In your BIOS make sure you disable Secure Boot. Not UEFI, not Legacy mode, specifically the switch that disables secure boot and nothing else. After these steps, WiFi works. To enter BIOS on Y50, tap F2 after Lenovo splash screen.

 

Edit 7.2.2017: WiFi performance is unfortunately ABYSMAL. Will update this post if I find any solutions. Connection is super slow and constantly dropping.

Luckily, USB tethering from Android works like a charm so it's not a deal breaker for now.

Edit 2: looks like blacklisting bcma driver makes things much much better:

sudo nano /etc/modprobe.d/blacklist.conf

#add this
blacklist bcma

sudo systemctl restart NetworkManager

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

Expose your dev machine to the public via reverse SSH tunnel

Scenario: you are creating a REST service which needs to be exposed to the public even in early stage of development due to an upstream provider which sends back feedback data from a webhook API.

You are also behind a NAT so you'd have to port forward yourself out but you can't do that for whatever reason. Or maybe you are behind a firewall and 7 proxies.

All you need is an external server with a public IP. Then, on your machine:

ssh -R 0.0.0.0:8080:localhost:8080 server_user@public_server_ip

In the above example, I used port 8080 which my REST service uses when I develop. On your public server, make sure you have

GatewayPorts yes

in /etc/ssh/sshd_config. If it is missing, add it.

And that's it.. your local REST service is now publicly accessible via public_server_ip:8080.

a-m-a-z-i-n-g

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

OJDBC7 in a Docker container? Prepare for trouble

Scenario: A JDK8 Docker container using OJDBC7 to connect to the database. Sounds simple enough, what could go wrong?

Simptoms: Connecting to the database randomly takes several minutes, fails with a weird SqlRecoverableException: no more data to read from socket or just works fine as if there is no problem.

The same Docker image also works fine on some machine but fails consistently on other.

The reason is this. Docker is not good at /dev/random. Probably even more so if you run it in a VM, since it's double isolated from actual entropy sources (my non scientific observation). For whatever reason, OJDBC defaults to /dev/random and this causes a block when connecting to the database due to high probability of /dev/random depletion.

Simple solution is to just mount /dev/urandom to /dev/random inside the Docker, in docker run command:

-v /dev/urandom:/dev/random

So.. if you ever want to use OJDBC inside Docker, remember this flag. It will save lives or at least spare you hours of useless debugging.

 

Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs

JPQL: getting whole entities on a distinct column

If you want distinct entities in JPQL you would normally write something like this:

SELECT DISTINCT(l) FROM Letter l WHERE l.name=:name;

But this will do a DISTINCT on the whole Entity. This will also fail on Oracle DB if your Entity contains a CLOB. What if you really want to do a DISTINCT on a field, for example:

SELECT DISTINCT(l.id) FROM Letter l WHERE l.name=:name;

Unfortunately, this only returns an array of ID fields. If you want to retrieve the full entities and do a DISTINCT on a field the final query looks like:

SELECT ll FROM Letter ll WHERE ll.id IN (SELECT DISTINCT(l.id) FROM Letter l WHERE l.name=:name);
Cen
GitHub
Eurobattle.net
Lagabuse.com
Bnetdocs