Accessing github behind a firewall

Some HPC clusters (e.g. SuperMUC-NG) restricts access to outside sources and thus does not allow connections to https servers. Nevertheless, GitHub can be used if remote port forwarding is correctly set. Here, we described the procedure to set up such port forwarding.

  1. On your local machine, add to your ~/.ssh/config the following lines:

Host supermucNG
   Hostname skx.supermuc.lrz.de
   User <Your Login>
   RemoteForward ddddd github.com:22

where ddddd is an arbitrary 5-digital port number, smaller than 65535.

  1. Use the following command to login onto the HPC cluster:

ssh supermucNG

Add the following lines to your ~/.ssh/config (on the HPC cluster):

Host github.com
   HostName localhost
   User git
   Port ddddd

With ddddd the same port number as before.

  1. Create SSH key by typing (use a non-empty passphrase, not too long as you will need to type it often)

ssh-keygen -t rsa

  1. Go to https://github.com/settings/ssh, add a new SSH key, and paste the public SSH key you just created (the content of ~/.ssh/id_rsa.pub on the HPC cluster).

  2. To allow cloning using ssh on SuperMUC-NG with the https address of git repository, add to ~/.gitconfig:

[url "git@github.com:"]
    insteadOf = https://github.com/

  1. (Optionnally) To avoid having to Enter your passphrase many times during the session, you can execute on the HPC cluster:

eval `ssh-agent -s`
ssh-add ~/.ssh/id_rsa

You should now be able to clone any GitHub repository, e.g. SeisSol including the submodules using:

git clone --recursive https://github.com/SeisSol/SeisSol.git

If it works, you will see several lines, for example:

Cloning into 'SeisSol'...
remote: Enumerating objects: 25806, done.
remote: Counting objects: 100% (4435/4435), done.
remote: Compressing objects: 100% (1820/1820), done.
remote: Total 25806 (delta 2972), reused 3710 (delta 2551), pack-reused 21371
Receiving objects: 100% (25806/25806), 110.50 MiB | 9.79 MiB/s, done.
Resolving deltas: 100% (19382/19382), done.

Accessing PyPI behind a firewall

Many post-processing scripts of SeisSol require Python dependencies. We describe how to use pip on a HPC cluster with restricts access to outside sources in the following.

  1. On your local machine in ~/.ssh/config add the following RemoteForward line:

Host supermucNG
    ...
    RemoteForward ddddd localhost:8899

where ddddd is an arbitrary port number with 5 digits. (This number should be different from port number used in other RemoteForward entries.)

  1. Install proxy.py on your local machine.

pip install --upgrade --user proxy.py

  1. Start proxy.py on your local machine. (And keep it running.)

~/.local/bin/proxy --port 8899

4. Login to the HPC cluster (e.g. ssh supermucNG). Check that you do not get: Warning: remote port forwarding failed for listen port ddddd. In this case you would need to change ddddd to a different port. Note that the problem might also be you have already an opened connection to the HPC cluster.

Once connected to the HPC cluster, pip can be used with

pip install <package name> --user --proxy http://localhost:ddddd/

where ddddd is your arbitrary port number.