ldap user search very slow

[dschick]

Hi,

my colleagues are little complaining about the fact, that the user search via ldap is pretty slow, even if one type a complete and correct user name.
I've checked with ldapsearch in the terminal, that the ldap connection itself, seems to be fast.

Is there anyway how to tackle this problem?

Best

Daniel

[johanfforsberg]

Hi Daniel,

I see, unfortunately I haven't currently got access to a LDAP server to test this against, although it was initially tested against a real one. Is it slower than when running without LDAP settings?

There is "throttling" of the search in the GUI, and it could be it's a little too much. It's set to not send search queries to the backend more often than every 0.5 seconds. This was intended to prevent overloading the LDAP, but perhaps I was being too careful. It would certainly also be possible to cache the results in the backend, since LDAP info rarely changes. If you're able to experiment with the frontend, you could maybe try changing "frontend/src/entryeditor.js", in the section

        this.slowFetchUserSuggestions = debounce(
            this.fetchUserSuggestions.bind(this),
            500
        );

you might want to experiment with changing the number 500 to something smaller, or even 0 and see what happens. It might send quite a lot of queries (one per key stroke I think) to the LDAP server though. If you try this, please report your results so we can change the setting.

Thanks for reporting this! BTW, did you try updating to the latest master? A change was merged to remember the last author list used, which might also make the situation less annoying in some cases.

[johanfforsberg]

To test the speed of the backend querying LDAP, you can also try something like

time curl http://<elogyhost>/api/users/?search=daniel

and see how long it takes. If it takes more than an instant, the problem is somewhere between the backend and the LDAP server.

Anyway, I've set up a LDAP server now so I can test, it does not seem very slow but the experience can definitely be improved :)

[dschick]

Thanks for your comments.
First of all the "remember-author" feature will pretty much solve most of our issues.

However, when I try the curl cmd I get a rather slow response

time curl http://elogy/api/users/?search=mylogin
{
    "users": [
        {
            "login": "mylogin",
            "name": "D......",
            "email": "D......"
        }
    ]
}

real	0m4,052s
user	0m0,012s
sys	0m0,011s

while an ldapsearch return quasi instantaneously from the same terminal.
Changing frontend throtteling seem not to be the issue here.

[johanfforsberg]

Wow, that's way slower than anything I've seen. I can see how that would be very annoying. It seems abnormal. I guess there is a problem either with the python ldap client, or somehow with the configuration. Or could there possibly be some kind of throttling going on at the ldap server side? It seems unlikely that there's some network issue causing four second delays... It would be really interesting if you could try doing the same query using the python ldap client (all the code is in backend/users.py) from the same machine where ldap search was fast. I'll look into caching the ldap lookups on the backend. It probably makes sense anyway.

…

On Mon, 11 Mar 2019, 12:30 Daniel Schick, ***@***.***> wrote: Thanks for your comments. First of all the "remember-author" feature will pretty much solve most of our issues. However, when I try the curl cmd I get a rather slow response time curl http://elogy/api/users/?search=mylogin { "users": [ { "login": "mylogin", "name": "D......", "email": "D......" } ] } real 0m4,052s user 0m0,012s sys 0m0,011s while an ldapsearch return quasi instantaneously from the same terminal. Changing frontend throtteling seem not to be the issue here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNy6EOJzcRWm8jO9TBwNhGQa28XoRmoks5vVj5sgaJpZM4bZXjf> .

[dschick]

hm, doing the ldap search directly in python is also pretty fast:
con.search_s("ou=people,o=server,dc=de", ldap.SCOPE_SUBTREE, filterstr="(|(cn=*sch*)(uid=*sch*))", attrlist=["uid", "cn", "mail"])

ähm, is the python lib doing a new binding on every search?
It does not seem to keep the binding to the ldap alive, right?
Maybe this is the problem? and also in case the server is throttling after to many rebindings

[dschick]

doint 100 rebinds and searches with python ldap takes 1.8sec and is perfectly linear with the number of attempts, so a single search takes 0.018sec.

[johanfforsberg]

I don't really know what the python client is doing under the hood, but I don't remember doing anything particularly to keep the session alive or anything so it's likely that this could be done better. But as you say, it does not seem to explain the issue anyway. Curious... I guess the final test would be to try the same thing from the elogy server, if you didn't already. If that's fast too I'm out of ideas... could it be related to running in docker..? Library versions? Some issue with the particular LDAP server you're using?

…

On Mon, 11 Mar 2019, 13:42 Daniel Schick, ***@***.***> wrote: doint 100 rebinds and searches with python ldap takes 1.8sec and is perfectly linear with the number of attempts, so a single search takes 0.018sec. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNy6GXjIr4upbdhnBqSsFwMW35AzOjDks5vVk9IgaJpZM4bZXjf> .

[dschick]

Indeed it's wired. Actually I test it from the elogy host. So it might be related to Docker or a different version of some packages. I tested it with python 2.7 Johan Forsberg <notifications@github.com> schrieb am Mo., 11. März 2019, 14:27:

…

I don't really know what the python client is doing under the hood, but I don't remember doing anything particularly to keep the session alive or anything so it's likely that this could be done better. But as you say, it does not seem to explain the issue anyway. Curious... I guess the final test would be to try the same thing from the elogy server, if you didn't already. If that's fast too I'm out of ideas... could it be related to running in docker..? Library versions? Some issue with the particular LDAP server you're using? On Mon, 11 Mar 2019, 13:42 Daniel Schick, ***@***.***> wrote: > doint 100 rebinds and searches with python ldap takes 1.8sec and is > perfectly linear with the number of attempts, so a single search takes > 0.018sec. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #13 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/ABNy6GXjIr4upbdhnBqSsFwMW35AzOjDks5vVk9IgaJpZM4bZXjf > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANKILNC2y5bjQeALV2Pm530dlBE-oU3xks5vVlmugaJpZM4bZXjf> .

[dschick]

So I could try to install elogy locally without Docker to check that, right? I fear that I would need some help/manual for that, as I was not able to follow the instructions of the Readme so far. Best Daniel Daniel Schick <schick.daniel@gmail.com> schrieb am Mo., 11. März 2019, 14:43:

…

Indeed it's wired. Actually I test it from the elogy host. So it might be related to Docker or a different version of some packages. I tested it with python 2.7 Johan Forsberg ***@***.***> schrieb am Mo., 11. März 2019, 14:27: > I don't really know what the python client is doing under the hood, but I > don't remember doing anything particularly to keep the session alive or > anything so it's likely that this could be done better. But as you say, it > does not seem to explain the issue anyway. Curious... > > I guess the final test would be to try the same thing from the elogy > server, if you didn't already. If that's fast too I'm out of ideas... > could > it be related to running in docker..? Library versions? Some issue with > the > particular LDAP server you're using? > > On Mon, 11 Mar 2019, 13:42 Daniel Schick, ***@***.***> > wrote: > > > doint 100 rebinds and searches with python ldap takes 1.8sec and is > > perfectly linear with the number of attempts, so a single search takes > > 0.018sec. > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #13 (comment) > >, > > or mute the thread > > < > https://github.com/notifications/unsubscribe-auth/ABNy6GXjIr4upbdhnBqSsFwMW35AzOjDks5vVk9IgaJpZM4bZXjf > > > > . > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#13 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ANKILNC2y5bjQeALV2Pm530dlBE-oU3xks5vVlmugaJpZM4bZXjf> > . >

[dschick]

Or could you easily test the difference in ldap performance between a local and a Docker installation? Daniel Schick <schick.daniel@gmail.com> schrieb am Mo., 11. März 2019, 17:09:

…

So I could try to install elogy locally without Docker to check that, right? I fear that I would need some help/manual for that, as I was not able to follow the instructions of the Readme so far. Best Daniel Daniel Schick ***@***.***> schrieb am Mo., 11. März 2019, 14:43: > Indeed it's wired. Actually I test it from the elogy host. > So it might be related to Docker or a different version of some packages. > I tested it with python 2.7 > > Johan Forsberg ***@***.***> schrieb am Mo., 11. März 2019, > 14:27: > >> I don't really know what the python client is doing under the hood, but I >> don't remember doing anything particularly to keep the session alive or >> anything so it's likely that this could be done better. But as you say, >> it >> does not seem to explain the issue anyway. Curious... >> >> I guess the final test would be to try the same thing from the elogy >> server, if you didn't already. If that's fast too I'm out of ideas... >> could >> it be related to running in docker..? Library versions? Some issue with >> the >> particular LDAP server you're using? >> >> On Mon, 11 Mar 2019, 13:42 Daniel Schick, ***@***.***> >> wrote: >> >> > doint 100 rebinds and searches with python ldap takes 1.8sec and is >> > perfectly linear with the number of attempts, so a single search takes >> > 0.018sec. >> > >> > — >> > You are receiving this because you commented. >> > Reply to this email directly, view it on GitHub >> > < >> #13 (comment) >> >, >> > or mute the thread >> > < >> https://github.com/notifications/unsubscribe-auth/ABNy6GXjIr4upbdhnBqSsFwMW35AzOjDks5vVk9IgaJpZM4bZXjf >> > >> > . >> > >> >> — >> You are receiving this because you authored the thread. >> Reply to this email directly, view it on GitHub >> <#13 (comment)>, >> or mute the thread >> <https://github.com/notifications/unsubscribe-auth/ANKILNC2y5bjQeALV2Pm530dlBE-oU3xks5vVlmugaJpZM4bZXjf> >> . >> >

[johanfforsberg]

I did test with docker, but only against my own test LDAP server with a few users. I'll do some more testing.

It might be that the search query we're using is too slow. I believe elogy being used with a real LDAP server at MAXIV, and as far as I can remember performance was OK when I was testing. But I suppose the performance could vary between server implementations, and maybe with database size.

The search string is e.g. (|(cn=*daniel*)(uid=*daniel*)), do you mind trying it with your ldap client?

Regarding local install, did you try the make install, make run way? If it does not work, I'd like to know what error you get. It will require python3.5 or later, though.

[dschick]

Hej,

I actually used the (|(cn=*daniel*)(uid=*daniel*)) query in my test with python from the elogy host and it was pretty fast.

I tried already the make install in my virtual machine for testing and it worked without problems.
However, I need to do a proper config, because I could not reach the webinterface so far.
Do I need to edit the config.py files for that?

For testing on my production machine, I would like to do also a clean uninstall of the local install. is that possible?

[dschick]

python2.7 vs python 3.6 does not make a difference

[johanfforsberg]

OK, so slow query is also eliminated... 🤔

The make scripts will start the backend on port 8888 and the frontend proxy on port 3000, if you run it with make -j2 run (that enables make to run both processes in parallel). So to check out the interface, go to port 3000 on your VM, and to talk directly to to the api you can use either port. You may have to make some changes to config.py though, at least to get LDAP to work. Just put in the LDAP variables you use in the docker installation.

Regarding uninstall, make install should not install anything outside the git checkout directory, and does not even need to be done as a super user. All python deps are installed in a virtualenv in backend/env and the npm stuff is in frontend/node_modules. Uninstalling elogy is just a matter of removing the git checkout.

The only thing you may have to do on the host is to install some of the dependencies in the underlying OS, in particular the ldap libs. E.g. sudo apt-get install libldap2-dev libsasl2-dev (on ubuntu).

[dschick]

now I am completely confused.
I just tried the ldap search from my elogy installation from within my virtual machine (only for testing) and it is pretty quick (instantaneous).

time curl http://<elogyhost>/api/users/?search=daniel

takes only 0.083sec instead of 4s on my production server.
Both systems are running now with docker with the latest source from github.
I did the same setup on both systems and also the OS of the virtual machine and the production server is the same.

The only obvious difference is, that both machines are in different networks.
However, the ldap search directly in python is a bit slower on the virtual machine than on the production server.