[X2Go-Dev] Bug#1342: Graphical apps stalling in suspended session

[yk+bug+]

Package: x2goserver
Version: 4.0.1.0

I put this in the x2goserver section but I'm not quite sure if it's the
right component that suffers from this issue.

In my company, we recently began to experience an issue similar to the
one reported in archived bug #673 [1] and on the X2Go-User list [2]
back in 2015. The manifestation of the issue is that when a session is
suspended, our program that runs in a terminal inside the session is
slowed down to a near stop, and abruptly recovers its original speed
when the session is resumed. There seems to be a link with graphical
components though, and my guess is it's the output scrolling in the
terminal that is stalled, and by way of consequence slowing down the
rest of the program downstream.
To confirm the involvement of the graphics, I tried running glxgears as
a test, and while it runs at about 1000 fps while the session is
attached, it drops at about 3 fps (!) when suspended. This behaviour is
reproducible on a freshly installed CentOS 7 VM; I can provide a
VirtualBox VM on which I ran the glxgears test.

What is strange is that we only noticed it a few days/weeks ago whereas
we didn't touch/update anything on the machines running those programs
and x2go. We have a cluster of 5 CentOS 7 machine + 1 CentOS 6 machine
and the issue appears on all of them, including CentOS 6, which hints
that the issue might not be related to a specific version of the x2go
components.

A quick search for this issue raised at least 2 third parties
experiencing the issue; one university [3] announced dropping x2go as
of October 2018 because of this, and another [4] is listing it as a
known issue, albeit maybe in 2015.

I hope we can find a source for this problem, it's quite critical for
us.

Here are versions of the installed x2go components on CentOS 7:
cups-x2go-3.0.1.3-1.el7.noarch
libNX_X11-3.5.99.16-1.el7.x86_64
nx-libs-3.5.99.16-1.el7.x86_64
nxagent-3.5.99.16-1.el7.x86_64
nxproxy-3.5.99.16-1.el7.x86_64
perl-X2Go-Log-4.1.0.0-1.el7.noarch
perl-X2Go-Server-4.1.0.0-1.el7.noarch
perl-X2Go-Server-DB-4.1.0.0-1.el7.x86_64
python-x2go-0.5.0.3-1.el7.noarch
x2goagent-4.1.0.0-1.el7.x86_64
x2goclient-4.1.1.1-1.el7.x86_64
x2godesktopsharing-3.1.1.2-1.el7.x86_64
x2goplugin-4.1.1.1-1.el7.x86_64
x2goplugin-provider-4.1.1.1-1.el7.x86_64
x2goserver-4.1.0.0-1.el7.x86_64
x2goserver-common-4.1.0.0-1.el7.noarch
x2goserver-fmbindings-4.1.0.0-1.el7.x86_64
x2goserver-printing-4.1.0.0-1.el7.x86_64
x2goserver-xsession-4.1.0.0-1.el7.noarch


[1] https://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=673
[2] http://lists.x2go.org/pipermail/x2go-user/2015-August/003397.html
[3] https://kb.thayer.dartmouth.edu/article/337-x2go
[4] https://www.cs.rutgers.edu/resources/accessing-computer-science-linux-desktop-using-x2go

[Ulrich Sibiller]

Post by yk+bug+
Package: x2goserver
Version: 4.0.1.0
I put this in the x2goserver section but I'm not quite sure if it's the
right component that suffers from this issue.
In my company, we recently began to experience an issue similar to the
one reported in archived bug #673 [1] and on the X2Go-User list [2]
back in 2015. The manifestation of the issue is that when a session is
suspended, our program that runs in a terminal inside the session is
slowed down to a near stop, and abruptly recovers its original speed
when the session is resumed. There seems to be a link with graphical
components though, and my guess is it's the output scrolling in the
terminal that is stalled, and by way of consequence slowing down the
rest of the program downstream.
To confirm the involvement of the graphics, I tried running glxgears as
a test, and while it runs at about 1000 fps while the session is
attached, it drops at about 3 fps (!) when suspended. This behaviour is
reproducible on a freshly installed CentOS 7 VM; I can provide a
VirtualBox VM on which I ran the glxgears test.

...

Post by yk+bug+
A quick search for this issue raised at least 2 third parties
experiencing the issue; one university [3] announced dropping x2go as
of October 2018 because of this, and another [4] is listing it as a
known issue, albeit maybe in 2015.

Oh, they should have asked before dropping X2go!

Post by yk+bug+
I hope we can find a source for this problem, it's quite critical for
us.

That has been implemented on purpose by nomachine to take some load
off the server. You can disable that via the sleep option:
sleep=<int>
delay X server operations when suspended (provided in
msec), set to 0 to keep nxagent session fully functional when
suspended (e.g. useful when mirroring an nxagent session via VNC)

You can set this sleep to 0 in /etc/x2go/x2goagent.options on the
server with the following line:
X2GO_NXAGENT_DEFAULT_OPTIONS="-options nx/nx,sleep=0"

This in non-obvious and should be a client option instead... Can you
please try if that helps for your setup?

Uli

[yk+bug+]

On Wed, 14 Nov 2018 21:50:59 +0100

Post by Ulrich Sibiller
That has been implemented on purpose by nomachine to take some load
sleep=<int>
delay X server operations when suspended (provided in
msec), set to 0 to keep nxagent session fully functional when
suspended (e.g. useful when mirroring an nxagent session via VNC)
You can set this sleep to 0 in /etc/x2go/x2goagent.options on the
X2GO_NXAGENT_DEFAULT_OPTIONS="-options nx/nx,sleep=0"
This in non-obvious and should be a client option instead... Can you
please try if that helps for your setup?

So I tried this options by adding it to the previous ones:
X2GO_NXAGENT_DEFAULT_OPTIONS+=" -options nx/nx,sleep=0"

It seems to solve indeed the problem at least for glxgears, which now
runs faster when I quit the x2go client, which is what I expect.
However, it has the serious side effect of preventing the sessions to
be suspended; the files clients, keyboard, state and the folders
mimebox and tools are not created in the x2go session folder
(/tmp/.x2go-user/<session>).
This side effect is kind of worse for us.

[Ulrich Sibiller]

Post by yk+bug+
On Wed, 14 Nov 2018 21:50:59 +0100

Post by Ulrich Sibiller
That has been implemented on purpose by nomachine to take some load
sleep=<int>
delay X server operations when suspended (provided in
msec), set to 0 to keep nxagent session fully functional when
suspended (e.g. useful when mirroring an nxagent session via VNC)
You can set this sleep to 0 in /etc/x2go/x2goagent.options on the
X2GO_NXAGENT_DEFAULT_OPTIONS="-options nx/nx,sleep=0"
This in non-obvious and should be a client option instead... Can you
please try if that helps for your setup?

X2GO_NXAGENT_DEFAULT_OPTIONS+=" -options nx/nx,sleep=0"

In my tests I have changed the _first_ entry in the options file, the
one without the +. Does that make any difference for you?

Post by yk+bug+
It seems to solve indeed the problem at least for glxgears, which now
runs faster when I quit the x2go client, which is what I expect.
However, it has the serious side effect of preventing the sessions to
be suspended; the files clients, keyboard, state and the folders

Quitting the x2goclient is effectively suspending the session since
there's no nxproxy anymore. You can see that in the file session.log
on the server (~(.x2go/C-username-something/...).

Providing that sleep=0 option is changing only one thing: it
eliminates any sleeps that have been implemented to take load off the
server in suspended state. There should not be any other behaviourial
differences.

Post by yk+bug+
mimebox and tools are not created in the x2go session folder
(/tmp/.x2go-user/<session>).
This side effect is kind of worse for us.

Can you please explain in detail? I am wondering what is happening here.

You can also try - instead of using the non-obvious setting in
x2goagent.setting - to adapt x2gostartagent on the server to integrate
that option into the generated options file.

Uli

[yk+bug+]

On Fri, 16 Nov 2018 09:54:51 +0100

Post by Ulrich Sibiller
In my tests I have changed the _first_ entry in the options file, the
one without the +. Does that make any difference for you?

It does not make any difference about the session suspension issue if I
either set "-options" alone (without the +=) or with any combination of
the other occurences in the file /etc/x2go/x2goagent.options

Post by Ulrich Sibiller

Post by yk+bug+
mimebox and tools are not created in the x2go session folder
(/tmp/.x2go-user/<session>).
This side effect is kind of worse for us.

Can you please explain in detail? I am wondering what is happening here.

The problem that I get when adding the/setting the -option switch is
that I cannot suspend the session any more (e.g. the
x2gosuspend-session command does not suspend the session) and I'm
getting the following line every 2 seconds in the log:

/usr/sbin/x2gocleansessions[1115]: <user>-50-1542963751_stDXFCE_dp32:
state file for this session does not
exist: /tmp/.x2go-<user>/C-<user>-50-1542963751_stDXFCE_dp32/state
(this can be ignored during session startups)

I also have some files missing as I stated previously (clients,
keyboard, state) which I think is linked somehow to the issue.

Post by Ulrich Sibiller
You can also try - instead of using the non-obvious setting in
x2goagent.setting - to adapt x2gostartagent on the server to integrate
that option into the generated options file.

Do you mean I should wrap a script around the x2gostartagent executable?
Or is there an configuration file somewhere that I can use?



On Sat, 17 Nov 2018 11:00:47 +0000

Post by Ulrich Sibiller
This is indeed peculiar. Are you sure, not syntax error is involved
in x2goagent.options. Note that the mimebox folder is used only by
Python X2Go, if you use X2Go Client, you can simply ignore its
non-existent.

I have only 2 lines uncommented on this file, and I checked several
times, so I don't think there is any syntax error.

Post by Ulrich Sibiller
For tracking this down, you need to provide some debug output of the
X2Go Server side.
Also, let us know, if you use X2Go Client or PyHoca-CLI or -GUI on
the client-side.
If you use PyHoca-*, run it with "--libdebug" on a cmd line and
provide the debug output.
If you use X2Go Client, you can use the --debug switch, but
beware... it will reveal user passwords, so don't send that unedited
to a public mailing list!

We are using the X2Go client, I will try to grab the debug version and
run it.



On Tue, 20 Nov 2018 15:45:01 -0700

Post by Ulrich Sibiller
See https://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=1322 for my
proposal for addressing this on the server side. I believe by adding
the line above you are messing up many of the default options.
-options filename
path to an options file containing nx/nx options (see
below).

Thanks for the suggestion. I barely have time to debug the issue, I'm
afraid having to patch and compile anything will be too time consuming.
I guess I'll wait for nightly builds if the patch is considered for
inclusion in the project.

[Mike Gabriel]

Hi,

Post by yk+bug+
On Wed, 14 Nov 2018 21:50:59 +0100

Post by Ulrich Sibiller
That has been implemented on purpose by nomachine to take some load
sleep=<int>
delay X server operations when suspended (provided in
msec), set to 0 to keep nxagent session fully functional when
suspended (e.g. useful when mirroring an nxagent session via VNC)
You can set this sleep to 0 in /etc/x2go/x2goagent.options on the
X2GO_NXAGENT_DEFAULT_OPTIONS="-options nx/nx,sleep=0"
This in non-obvious and should be a client option instead... Can you
please try if that helps for your setup?

X2GO_NXAGENT_DEFAULT_OPTIONS+=" -options nx/nx,sleep=0"

ok...

Post by yk+bug+
It seems to solve indeed the problem at least for glxgears, which now
runs faster when I quit the x2go client, which is what I expect.
However, it has the serious side effect of preventing the sessions to
be suspended; the files clients, keyboard, state and the folders
mimebox and tools are not created in the x2go session folder
(/tmp/.x2go-user/<session>).
This side effect is kind of worse for us.

This is indeed peculiar. Are you sure, not syntax error is involved
in x2goagent.options. Note that the mimebox folder is used only by
Python X2Go, if you use X2Go Client, you can simply ignore its
non-existent.

For tracking this down, you need to provide some debug output of the
X2Go Server side.

Also, let us know, if you use X2Go Client or PyHoca-CLI or -GUI on the
client-side.

If you use PyHoca-*, run it with "--libdebug" on a cmd line and
provide the debug output.

If you use X2Go Client, you can use the --debug switch, but beware...
it will reveal user passwords, so don't send that unedited to a public
mailing list!

Mike

[Stefan Baur]

If you use X2Go Client, you can use the --debug switch, but beware... it
will reveal user passwords, so don't send that unedited to a public
mailing list!

This only happens in broker mode (and probably only if you use the http
broker, ssh broker mode *should* be fine - but please test and check).
Regular client-server use is unaffected.

-Stefan

[Orion Poplawski]

Post by Ulrich Sibiller

Post by yk+bug+
Package: x2goserver
Version: 4.0.1.0
I put this in the x2goserver section but I'm not quite sure if it's the
right component that suffers from this issue.
In my company, we recently began to experience an issue similar to the
one reported in archived bug #673 [1] and on the X2Go-User list [2]
back in 2015. The manifestation of the issue is that when a session is
suspended, our program that runs in a terminal inside the session is
slowed down to a near stop, and abruptly recovers its original speed
when the session is resumed. There seems to be a link with graphical
components though, and my guess is it's the output scrolling in the
terminal that is stalled, and by way of consequence slowing down the
rest of the program downstream.
To confirm the involvement of the graphics, I tried running glxgears as
a test, and while it runs at about 1000 fps while the session is
attached, it drops at about 3 fps (!) when suspended. This behaviour is
reproducible on a freshly installed CentOS 7 VM; I can provide a
VirtualBox VM on which I ran the glxgears test.

...

Post by yk+bug+
A quick search for this issue raised at least 2 third parties
experiencing the issue; one university [3] announced dropping x2go as
of October 2018 because of this, and another [4] is listing it as a
known issue, albeit maybe in 2015.

Oh, they should have asked before dropping X2go!

Post by yk+bug+
I hope we can find a source for this problem, it's quite critical for
us.

That has been implemented on purpose by nomachine to take some load
sleep=<int>
delay X server operations when suspended (provided in
msec), set to 0 to keep nxagent session fully functional when
suspended (e.g. useful when mirroring an nxagent session via VNC)
You can set this sleep to 0 in /etc/x2go/x2goagent.options on the
X2GO_NXAGENT_DEFAULT_OPTIONS="-options nx/nx,sleep=0"
This in non-obvious and should be a client option instead... Can you
please try if that helps for your setup?
Uli

See https://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=1322 for my proposal for
addressing this on the server side. I believe by adding the line above you
are messing up many of the default options. According to the man page of nxagent:

-options filename
path to an options file containing nx/nx options (see below).

[Ulrich Sibiller]

Post by Orion Poplawski
See https://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=1322 for my proposal for
addressing this on the server side. I believe by adding the line above you
-options filename
path to an options file containing nx/nx options (see below).

Yes, I know. I have worked on that option handling stuff in nxagent
several times. But x2go is not passing the options file via the
command line option. It is embedding the path into the DISPLAY
variable. My (quick) tests showed that it worked. I am still waiting
for the original poster to specify in detail what problems he had
after applying that configuration. Nevertheless I have not looked into
the code to check if there's anything that will prevent this from
working properly.

Regarding your proposal: While it is not the cleanest approach I think
we should include it to help users in special cases.. The sleep stuff
really should be handled by x2goclient. @Mihai Can you please include
the patch for the next release?

Uli

[Mike Gabriel]

Hi Uli, hi Orion,

Post by Ulrich Sibiller

Post by Orion Poplawski
See https://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=1322 for my proposal for
addressing this on the server side. I believe by adding the line above you
-options filename
path to an options file containing nx/nx options (see below).

Yes, I know. I have worked on that option handling stuff in nxagent
several times. But x2go is not passing the options file via the
command line option. It is embedding the path into the DISPLAY
variable. My (quick) tests showed that it worked. I am still waiting
for the original poster to specify in detail what problems he had
after applying that configuration. Nevertheless I have not looked into
the code to check if there's anything that will prevent this from
working properly.
Regarding your proposal: While it is not the cleanest approach I think
we should include it to help users in special cases.. The sleep stuff
the patch for the next release?

I have just applied Orion's patch from #1322.

Nightly build of x2goserver is on its way...

Mike