V1.9.0 published

HKilic · June 7, 2023, 8:18pm

I have a theory. I am now testing. It will be ready for test in shortly

HKilic · June 8, 2023, 4:39pm

I fixed the bug. just update docker images with below commands

docker pull ferrumgate/rest.portal:1.9.0
docker pull ferrumgate/job.admin:1.9.0
ferrumgate --restart

Also we upgraded client version to 1.6.0

Mart · June 9, 2023, 12:08am

I updated the server but just want to bring out that the Arm version of the Mac client is still complaining that it’s damaged. I don’t get that warning with the x64 version.

Running the command in Terminal is fixing it : xattr -rc /Applications/FerrumGate.app

Mart · June 9, 2023, 12:32am

Since I did the update on the server it’s worst now. I get ping error and reconnection of the tunnel after a few seconds. I tried on Windows and Mac with client 1.5 and 1.6 same issue :

[2023-06-08 20:29:44.818] [error] Error: ping failed
[2023-06-08 20:29:47.738] [info] executing command at worker
[2023-06-08 20:29:47.739] [info] sync network status 2023-06-09T00:29:47.738Z
[2023-06-08 20:29:47.739] [info] sync network status [{“id”:“PzWPXI1r7xDCt5t9”,“name”:“aws”,“action”:“allow”,“sshHost”:“ztna.mydomain.com:9999”,“tunnel”:{“tryCount”:0,“lastTryTime”:1686270580125,“isWorking”:true,“resolvErrorCount”:1,“resolvTimes”:[3000],“lastError”:“”,“assignedIp”:“100.64.0.9”,“serviceNetwork”:“172.28.28.0/24”,“resolvIp”:“172.28.28.1”,“resolvSearch”:“aws.mydomain.zero”,“tun”:“ferrumTEDetD”,“isMasterResolv”:true,“resolvTunDomains”:[]}}]
[2023-06-08 20:29:47.818] [error] Error: ping failed
[2023-06-08 20:29:49.674] [info] get tokens

Mart · June 9, 2023, 1:16am

I just want to bring out that it seems to be the default tunnel now that’s having issue with ping. The second tunnel on port 9998 is fine. When I have both connected only the default one on port 9999 is reconnecting all the time, the second one on port 9998 is stable. When I try connecting only 1 tunnel at a time, the one on port 9999 will reconnect after a few seconds but the one on port 9998 is fine. I’ll try creating a third one just to see if it’s related to the default one with the latest update.

I created a new tunnel on port 9997 and it’s working fine. Now I have both tunnels 9998 and 9997 connected and everything seems fine.

I saw that I can reassign services to the new network but I saw that the Assigned IP doesn’t change when I move the service to another network so I will have to recreate those I guess.

HKilic · June 9, 2023, 6:31am

xattr -rc /Applications/FerrumGate.app
I searched a little but could not find why, I delay to solve this one.

I guess there are some bugs about moving services to other networks.

Network at 9999 is strange,
please create a new network at 9996 and test 3 networks
9998 9997 and 9996,

please let me know the results.

Mart · June 21, 2023, 7:25pm

I just want to report that when you have more than 100 services define you see 1_ instead of the real number. It doesn’t create any issue but I just want to bring it out. If I filter per Networks and the one I select as less than 100 services it’s fine.

Capture d’écran, le 2023-06-21 à 15.24.00

It’s also doing the same thing in the Authorization section

Capture d’écran, le 2023-06-21 à 15.44.47

HKilic · June 21, 2023, 7:49pm

Added to task list. Thanks so much

Mart · June 22, 2023, 12:45pm

I just want to bring out that since I stopped using the default tunnel on port 9999 it’s working really well for everyone. I have 2 tunnels connected all the time and I also have 10 users testing and everyone seems happy so far. Like I reported yesterday I now have over a 100 services define in 2 different networks and I can connect to each of them fine.

I saw twice that for an unknown reason the DNS will remain define on the Wifi connection on the macOS client when Ferrumgate client is disconnected but removing it manually bring back internet access. As I have it connected most of the time it’s hard to see what is causing that. If I can find a pattern that create it I’ll let you know.

Mart · June 22, 2023, 5:23pm

Is there a limitation on the number of policies I can add to a network? I just started having a weird issue where I can create the service, assign an authorization policy to it but it won’t work. I delete it, create the same service in another network where I have less than 100 services and it’s fine.

I found a way to bypass it for now. I created a third network so now I have one tunnel on port 9998, another one on port 9997 and a third one on port 9996. Since I’m creating services for different location I’m fine doing that but just wondering if you are aware of a limitation?
Capture d’écran, le 2023-06-22 à 13.12.54

HKilic · June 22, 2023, 7:30pm

I am so glad that everything works well.
We will check DNS trouble on MacOS, I hope we can find it.
100 plus services is also amazing. thanks for it.

HKilic · June 22, 2023, 7:32pm

There is no limit about any thing. please check job.admin logs, one of its responsible is policy management.

HKilic · June 24, 2023, 10:43pm

Probably there is a bug about, so much container inspection. trying to simulate them

Mart · June 26, 2023, 1:18pm

Where that log would be stored? I’m not really familiar with docker so I know the containers are located in /var/lib/docker but that’s about it so I don’t know where I need to look to get the logs it doesn’t seem to be in /var/logs

Today the tunnel is unstable for everyone I tried rebooting the server, no difference. It disconnect and reconnect all the time. I’m trying to locate the log files to see if that could give me an idea why it’s unstable. Only the tunnel that has over 100 services is unstable the 2 others are fine.

Ok I think I found how to list job.admin but it seems there’s an issue with the dns server I see a bunch of undefined :

[2023-06-26T12:46:25.434] [INFO] default - read local dns /local/dns/rdp.clientserver10.aws.mydomain.zero/a → undefined

[2023-06-26T12:46:25.435] [INFO] default - read local dns /local/dns/rdp.client01.aws.mydomain.zero/a → undefined

[2023-06-26T12:46:25.435] [INFO] default - read local dns /local/dns/rdp.serverus1.awsusa.mydomain.zero/a → undefined

[2023-06-26T12:46:25.435] [INFO] default - read local dns /local/dns/mysql.server3.aws.mydomain.zero/a → undefined

[2023-06-26T12:46:25.435] [INFO] default - read local dns /local/dns/https.server1.aws.mydomain.zero/a → undefined

Just saw this error when I tried stopping ferrumgate :

Failed to allocate directory watch: Too many open files

I just tried doing this :
sudo sysctl fs.inotify.max_user_instances=512

I don’t know if that will make a difference.

HKilic · June 26, 2023, 2:05pm

please paste 3 lines below.

IDS=$(docker ps|grep admin|cut -d" " -f1)
for id in $IDS; do docker logs $id |grep ERROR; done;
for id in $IDS; do docker logs $id |grep “Too many open” -A10 -B10; done;

please paste output.

also for memory and cpu
please type 3 below commands and outputs

apt install sysstat
free
iostat

HKilic · June 26, 2023, 2:07pm

probably after opening 100 services, there is a limit error about “open files”

sudo sysctl fs.inotify.max_user_instances=512

this will not work for docker containers.

Mart · June 26, 2023, 3:41pm

This is what I get with the second line :

[2023-06-26T13:53:03.260] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:53:03.292] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:53:03.295] [ERROR] default - current gateway not found luxnn2r5fyuke3fo

[2023-06-26T13:53:03.512] [ERROR] default - current gateway not found luxnn2r5fyuke3fo

[2023-06-26T13:53:04.111] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:53:04.219] [ERROR] default - ConnectionError: connect ECONNREFUSED 10.10.6.8:9200

[2023-06-26T15:18:04.074] [ERROR] default - Error: Command failed: ip link set dev ferrumlsYulgK8 up

[2023-06-26T13:53:01.141] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:53:01.181] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:53:01.205] [ERROR] default - current gateway not found i5x3i56tkixvqwge

[2023-06-26T13:53:01.734] [ERROR] default - current gateway not found i5x3i56tkixvqwge

[2023-06-26T13:53:02.411] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:53:02.577] [ERROR] default - ConnectionError: connect ECONNREFUSED 10.10.6.8:9200

[2023-06-26T13:52:57.373] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:52:57.439] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:52:57.461] [ERROR] default - current gateway not found a2gr7he353ygvrvu

[2023-06-26T13:52:58.011] [ERROR] default - current gateway not found a2gr7he353ygvrvu

[2023-06-26T13:52:58.772] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:52:59.122] [ERROR] default - ConnectionError: connect ECONNREFUSED 10.10.6.8:9200

[2023-06-26T13:53:27.618] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:53:28.183] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:53:57.619] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:53:58.183] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:54:27.618] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:54:28.183] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:54:57.617] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:54:58.184] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:55:27.616] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:55:28.184] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:55:57.615] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:55:58.184] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:56:27.614] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:56:28.185] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:56:57.613] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:56:58.185] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:57:27.612] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:57:28.185] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:57:57.620] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:57:58.186] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:58:27.624] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:58:28.187] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:58:57.628] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:58:58.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:59:27.628] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:59:28.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:59:57.632] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:59:58.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:00:27.636] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:00:28.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:00:57.644] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:00:58.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:01:27.648] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:01:28.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:01:57.648] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:01:58.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:02:27.652] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:02:28.188] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:02:57.656] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:02:58.189] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:03:27.656] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:03:28.189] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:03:57.660] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:03:58.190] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:04:27.664] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:04:28.190] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:04:57.664] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:04:58.190] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:05:27.672] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:05:28.191] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:05:57.676] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:05:58.190] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:06:27.680] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:06:28.191] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:06:57.680] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:06:58.190] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:07:27.684] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:07:28.191] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:07:57.692] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:07:58.192] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:08:27.692] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:08:28.192] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:08:57.693] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:08:58.192] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:09:27.692] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:09:28.192] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:09:57.693] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:09:58.193] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:10:27.696] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:10:28.193] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:10:57.695] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:10:58.194] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:11:27.696] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:11:28.193] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:11:57.696] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:11:58.193] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:12:27.700] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:12:28.194] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:12:57.700] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:12:58.194] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:13:27.700] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:13:28.194] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:13:57.704] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:13:58.194] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:14:27.708] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:14:28.195] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:14:57.713] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:14:58.195] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:15:27.711] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:15:28.195] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:15:57.712] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:15:58.195] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:16:27.712] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:16:28.195] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:16:57.712] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:16:58.195] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:17:27.712] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:17:28.196] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:17:57.712] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:17:58.196] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:18:27.712] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:18:28.195] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T14:18:57.712] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T15:37:58.221] [ERROR] default - current network disabled for gateway a2gr7he353ygvrvu

[2023-06-26T13:52:54.090] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:52:54.156] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:52:54.181] [ERROR] default - current gateway not found 7iynjdedsgr78ole

[2023-06-26T13:52:54.817] [ERROR] default - current gateway not found 7iynjdedsgr78ole

[2023-06-26T13:52:55.601] [ERROR] default - RestfullException [Error]: config is not ready

[2023-06-26T13:52:55.821] [ERROR] default - ConnectionError: connect ECONNREFUSED 10.10.6.8:9200

Mart · June 26, 2023, 3:42pm

For the third line I only get this :

grep: many: No such file or directory

grep: open”: No such file or directory

grep: many: No such file or directory

grep: open”: No such file or directory

grep: many: No such file or directory

Mart · June 26, 2023, 3:44pm

I get this using free :

And this using iostat :

Capture d’écran, le 2023-06-26 à 11.43.34

HKilic · June 26, 2023, 4:29pm

please type these commands, I need to find which part gives Too many file error

IDS=$(docker ps|grep admin|cut -d" " -f1)
for id in $IDS; do docker logs $id |grep "Too many open" -A10 -B10; done;

IDS=$(docker ps|grep ssh|cut -d" " -f1)
for id in $IDS; do docker logs $id |grep "Too many open" -A10 -B10; done;