Unique Mode is Very, Very Slow

[alpiccioni]

Expected Behavior

Unique mode can process at least ~500 QPS, with preference for >2000 QPS.

Actual Behavior

Unique mode processes ~200 QPS in the optimal case.

Steps to Reproduce the Problem

1. Press Ctrl-U on the verbose stream.

Specifications

• Version: Purr: 1.3.0, fzf: 0.40.0 (fb76893), zsh: 5.9
• Platform: 6.3.11-1rodete1-amd64

Problem

This is a complex problem that has to do with the efficiency of what purr is doing on unique mode. There's a lot of solutions to this problem, but all are more complex than what is currently being done by quite a bit. I need to think about how I can fix this elegantly without compiling anything if possible. Massive multi-threading may be the solution.

[questor]

this mode (ctrl-u) is also not mentioned in the readme nor in the preview-help....

[alpiccioni]

Hey! This is actually intentional for now. Until I get both the bugs filed for this fixed, I'm hesitant to advertise this in the help, as it's really more of an in-development feature at this point.

Thanks for pointing it out, though! I had forgotten that I excluded it and I need to remember to add it when I get it fully done.

[alpiccioni]

Massive parallelization is hard here because each streams needs the messages to come out in order. Unfortunately, just running the messages through a while loop in zsh with tail is pretty slow at about 1000QPS in Linux. That effectively caps how quickly we can process each message, and the overhead on each pipe IO operation pushing between threads is also very slow; 1ms per query is prohibitive.

There's a bunch of fancy solutions I can do here if I want to keep this pure zsh. Thread coordination is hard, but it's not impossible. But I think that's not required; I'm hammering screws at this point.

We already declare a dependency on perl. I think the best thing to do is to bundle a trie implementation in perl and do all the processing in a sub-process running perl, which should be much much faster to begin with, and perl is pretty good at text manipulation and string comparison to begin with.