Fixing Presence Conflicts #3320

jjj333-p · 2024-02-06T05:54:11Z

This is meant to cache client presence for a moment so that it doesn't oscillate.

Currently Dendrite just federates out whatever presence it gets from the sync loop, which means if theres any clients attempting to sync without setting the user online, and there is an online client, it will just flip back and forth each time one of the clients polls /sync.

This pull request essentially stores in a map when the client last set online ideally to allow the online client to sync again and set an online presence before setting idle or offline.

I am not great at programming nor am I familiar with this codebase so if this pr is just shitwater feel free to discard, just trying to fix an issue that severely bothers me. If it is easier you can also steal the code and write it in yourself. I ran the linter, not sure that it did anything, the vscode go extension seems to format and lint anyways.

I tried to run unit tests but I have no idea any of this thing. it errors on TestRequestPool_updatePresence/same_presence_is_not_published_dummy2 (10m0s) which I think making this change broke. I am unsure how to comply, if y'all point me in the right direction ill try to fix it. I have tested it with all the situations I can think of on my personal instance pain.agency, and this seems to stand up under all the previously bugged situations.

~~My go also decided to update a bunch of the dependencies, I hate git and github and have no idea how to fix that, it was not intentional.~~ i just overwrote them with the ones from the main repo and committed it, seems to have done what was needed.

Pull Request Checklist

I have added Go unit tests or Complement integration tests for this PR or I have justified why this PR doesn't need tests
Pull request includes a sign off below using a legally identifiable name or I have already signed off privately

Signed-off-by: Joseph Winkie <jjj333.p.1325@gmail.com>

This reverts commit a4c4fb9.

jjj333-p · 2024-02-06T05:55:01Z

heres the unit test full output:

joseph@f ~/jendrite (presence-optimization) [1]> DENDRITE_TEST_SKIP_NODB=1 go test --race ./...
# github.com/matrix-org/dendrite/internal/sqlutil
internal/sqlutil/sqlutil_test.go:10:2: missing go.sum entry for module providing package github.com/DATA-DOG/go-sqlmock (imported by github.com/matrix-org/dendrite/internal/sqlutil); to add:
        go get -t github.com/matrix-org/dendrite/internal/sqlutil
FAIL    github.com/matrix-org/dendrite/internal/sqlutil [setup failed]
# maunium.net/go/mautrix/crypto/olm
../go/pkg/mod/maunium.net/go/mautrix@v0.17.0/crypto/olm/account.go:6:11: fatal error: olm/olm.h: No such file or directory
    6 | // #include <olm/olm.h>
      |           ^~~~~~~~~~~
compilation terminated.
?       github.com/matrix-org/dendrite/appservice/api   [no test files]
?       github.com/matrix-org/dendrite/appservice/consumers     [no test files]
?       github.com/matrix-org/dendrite/appservice/query [no test files]
?       github.com/matrix-org/dendrite/build/dendritejs-pinecone        [no test files]
?       github.com/matrix-org/dendrite/build/gobind-yggdrasil   [no test files]
?       github.com/matrix-org/dendrite/clientapi/api    [no test files]
?       github.com/matrix-org/dendrite/clientapi/auth/authtypes [no test files]
?       github.com/matrix-org/dendrite/clientapi/httputil       [no test files]
?       github.com/matrix-org/dendrite/clientapi/producers      [no test files]
?       github.com/matrix-org/dendrite/clientapi/threepid       [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone       [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/conn  [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/defaults      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/embed [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/monolith      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/rooms [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/users [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/embed        [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/signing      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/yggconn      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/yggrooms     [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-upgrade-tests       [no test files]
?       github.com/matrix-org/dendrite/cmd/furl [no test files]
?       github.com/matrix-org/dendrite/cmd/generate-config      [no test files]
?       github.com/matrix-org/dendrite/cmd/generate-keys        [no test files]
?       github.com/matrix-org/dendrite/cmd/resolve-state        [no test files]
?       github.com/matrix-org/dendrite/federationapi/api        [no test files]
?       github.com/matrix-org/dendrite/federationapi/producers  [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/cache      [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/postgres   [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/postgres/deltas    [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/shared     [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/shared/receipt     [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/sqlite3    [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/sqlite3/deltas     [no test files]
?       github.com/matrix-org/dendrite/federationapi/types      [no test files]
?       github.com/matrix-org/dendrite/internal/eventutil       [no test files]
?       github.com/matrix-org/dendrite/internal/hooks   [no test files]
?       github.com/matrix-org/dendrite/mediaapi [no test files]
?       github.com/matrix-org/dendrite/mediaapi/fileutils       [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/postgres        [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/shared  [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/sqlite3 [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/tables  [no test files]
?       github.com/matrix-org/dendrite/mediaapi/thumbnailer     [no test files]
?       github.com/matrix-org/dendrite/mediaapi/types   [no test files]
?       github.com/matrix-org/dendrite/relayapi/api     [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage/postgres        [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage/shared  [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage/sqlite3 [no test files]
ok      github.com/matrix-org/dendrite/appservice       3.425s
ok      github.com/matrix-org/dendrite/build/gobind-pinecone    2.418s
FAIL    github.com/matrix-org/dendrite/clientapi [build failed]
ok      github.com/matrix-org/dendrite/clientapi/auth   (cached)
ok      github.com/matrix-org/dendrite/clientapi/routing        (cached)
ok      github.com/matrix-org/dendrite/clientapi/userutil       (cached)
ok      github.com/matrix-org/dendrite/cmd/create-account       1.079s
ok      github.com/matrix-org/dendrite/cmd/dendrite     1.123s
ok      github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/conduit       (cached)
ok      github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/relay (cached)
ok      github.com/matrix-org/dendrite/federationapi    (cached)
ok      github.com/matrix-org/dendrite/federationapi/consumers  (cached)
ok      github.com/matrix-org/dendrite/federationapi/internal   (cached)
ok      github.com/matrix-org/dendrite/federationapi/queue      (cached)
ok      github.com/matrix-org/dendrite/federationapi/routing    (cached)
ok      github.com/matrix-org/dendrite/federationapi/statistics (cached)
ok      github.com/matrix-org/dendrite/federationapi/storage    (cached)
ok      github.com/matrix-org/dendrite/federationapi/storage/tables     (cached)
ok      github.com/matrix-org/dendrite/internal (cached)
ok      github.com/matrix-org/dendrite/internal/caching (cached)
ok      github.com/matrix-org/dendrite/internal/fulltext        (cached)
ok      github.com/matrix-org/dendrite/internal/httputil        (cached)
ok      github.com/matrix-org/dendrite/internal/pushgateway     (cached)
ok      github.com/matrix-org/dendrite/internal/pushrules       (cached)
ok      github.com/matrix-org/dendrite/internal/transactions    (cached)
ok      github.com/matrix-org/dendrite/mediaapi/routing (cached)
ok      github.com/matrix-org/dendrite/mediaapi/storage (cached)
ok      github.com/matrix-org/dendrite/relayapi (cached)
ok      github.com/matrix-org/dendrite/relayapi/internal        (cached)
ok      github.com/matrix-org/dendrite/relayapi/routing (cached)
ok      github.com/matrix-org/dendrite/relayapi/storage/tables  (cached)
?       github.com/matrix-org/dendrite/roomserver/internal      [no test files]
?       github.com/matrix-org/dendrite/roomserver/internal/perform      [no test files]
?       github.com/matrix-org/dendrite/roomserver/producers     [no test files]
?       github.com/matrix-org/dendrite/roomserver/storage       [no test files]
?       github.com/matrix-org/dendrite/roomserver/storage/postgres      [no test files]
?       github.com/matrix-org/dendrite/roomserver/storage/sqlite3       [no test files]
?       github.com/matrix-org/dendrite/roomserver/version       [no test files]
?       github.com/matrix-org/dendrite/setup    [no test files]
?       github.com/matrix-org/dendrite/setup/jetstream  [no test files]
?       github.com/matrix-org/dendrite/setup/mscs       [no test files]
?       github.com/matrix-org/dendrite/setup/process    [no test files]
?       github.com/matrix-org/dendrite/syncapi/consumers        [no test files]
?       github.com/matrix-org/dendrite/syncapi/producers        [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/postgres [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/postgres/deltas  [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/sqlite3  [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/sqlite3/deltas   [no test files]
?       github.com/matrix-org/dendrite/syncapi/streams  [no test files]
?       github.com/matrix-org/dendrite/test     [no test files]
?       github.com/matrix-org/dendrite/test/testrig     [no test files]
?       github.com/matrix-org/dendrite/userapi/api      [no test files]
?       github.com/matrix-org/dendrite/userapi/producers        [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/postgres [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/postgres/deltas  [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/shared   [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/sqlite3  [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/sqlite3/deltas   [no test files]
?       github.com/matrix-org/dendrite/userapi/types    [no test files]
ok      github.com/matrix-org/dendrite/roomserver       6.391s
ok      github.com/matrix-org/dendrite/roomserver/acls  (cached)
ok      github.com/matrix-org/dendrite/roomserver/api   (cached)
ok      github.com/matrix-org/dendrite/roomserver/auth  (cached)
ok      github.com/matrix-org/dendrite/roomserver/internal/helpers      (cached)
ok      github.com/matrix-org/dendrite/roomserver/internal/input        (cached)
ok      github.com/matrix-org/dendrite/roomserver/internal/query        (cached)
ok      github.com/matrix-org/dendrite/roomserver/state (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/postgres/deltas       (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/shared        (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas        (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/tables        (cached)
ok      github.com/matrix-org/dendrite/roomserver/types (cached)
ok      github.com/matrix-org/dendrite/setup/base       (cached)
ok      github.com/matrix-org/dendrite/setup/config     (cached)
ok      github.com/matrix-org/dendrite/setup/mscs/msc2836       (cached)
ok      github.com/matrix-org/dendrite/syncapi  4.661s
ok      github.com/matrix-org/dendrite/syncapi/internal (cached)
ok      github.com/matrix-org/dendrite/syncapi/notifier (cached)
ok      github.com/matrix-org/dendrite/syncapi/routing  1.168s
ok      github.com/matrix-org/dendrite/syncapi/storage  (cached)
ok      github.com/matrix-org/dendrite/syncapi/storage/shared   (cached)
ok      github.com/matrix-org/dendrite/syncapi/storage/tables   (cached)
panic: test timed out after 10m0s
running tests:
        TestRequestPool_updatePresence (10m0s)
        TestRequestPool_updatePresence/same_presence_is_not_published_dummy2 (10m0s)

goroutine 620 [running]:
testing.(*M).startAlarm.func1()
        /usr/lib/golang/src/testing/testing.go:2259 +0x259
created by time.goFunc
        /usr/lib/golang/src/time/sleep.go:176 +0x45

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000007a00, {0xef5917, 0x1e}, 0xf2ddb0)
        /usr/lib/golang/src/testing/testing.go:1649 +0x871
testing.runTests.func1(0x0?)
        /usr/lib/golang/src/testing/testing.go:2054 +0x85
testing.tRunner(0xc000007a00, 0xc0002b1af8)
        /usr/lib/golang/src/testing/testing.go:1595 +0x262
testing.runTests(0xc0000b7720?, {0x13ee6e0, 0x1, 0x1}, {0x1c?, 0x1e?, 0x1459fe0?})
        /usr/lib/golang/src/testing/testing.go:2052 +0x8ae
testing.(*M).Run(0xc0000b7720)
        /usr/lib/golang/src/testing/testing.go:1925 +0xcd8
main.main()
        _testmain.go:47 +0x2be

goroutine 19 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
        /home/joseph/go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:882 +0x8b
created by github.com/golang/glog.init.0 in goroutine 1
        /home/joseph/go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:410 +0x258

goroutine 35 [sleep]:
time.Sleep(0x2faf080)
        /usr/lib/golang/src/runtime/time.go:195 +0x125
github.com/matrix-org/dendrite/syncapi/sync.(*RequestPool).cleanPresence(0xc000230000, {0xffd378?, 0x23c0620}, 0x2faf080)
        /home/joseph/jendrite/syncapi/sync/requestpool.go:119 +0x152
created by github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence in goroutine 6
        /home/joseph/jendrite/syncapi/sync/requestpool_test.go:129 +0x45a

goroutine 6 [chan receive]:
testing.(*T).Run(0xc000007ba0, {0xefb650, 0x25}, 0xc0002281a0)
        /usr/lib/golang/src/testing/testing.go:1649 +0x871
github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence(0x0?)
        /home/joseph/jendrite/syncapi/sync/requestpool_test.go:131 +0x49b
testing.tRunner(0xc000007ba0, 0xf2ddb0)
        /usr/lib/golang/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 1
        /usr/lib/golang/src/testing/testing.go:1648 +0x846

goroutine 40 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x12eb090?, 0x80?, 0xc00022e540?)
        /usr/lib/golang/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc00022c010)
        /usr/lib/golang/src/sync/mutex.go:171 +0x213
sync.(*Mutex).Lock(0xc00022c010)
        /usr/lib/golang/src/sync/mutex.go:90 +0x55
github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence.func1(0xc0002269c0)
        /home/joseph/jendrite/syncapi/sync/requestpool_test.go:132 +0x59
testing.tRunner(0xc0002269c0, 0xc0002281a0)
        /usr/lib/golang/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 6
        /usr/lib/golang/src/testing/testing.go:1648 +0x846
FAIL    github.com/matrix-org/dendrite/syncapi/sync     600.041s
ok      github.com/matrix-org/dendrite/syncapi/synctypes        (cached)
ok      github.com/matrix-org/dendrite/syncapi/types    (cached)
ok      github.com/matrix-org/dendrite/userapi  (cached)
ok      github.com/matrix-org/dendrite/userapi/consumers        (cached)
ok      github.com/matrix-org/dendrite/userapi/internal (cached)
ok      github.com/matrix-org/dendrite/userapi/storage  (cached)
ok      github.com/matrix-org/dendrite/userapi/storage/tables   (cached)
ok      github.com/matrix-org/dendrite/userapi/util     (cached)
FAIL
⏎                                                                                                                                                                                                                        
joseph@f ~/jendrite (presence-optimization) [1]>

jjj333-p · 2024-02-06T05:56:50Z

Oh I should also mention don't look too closely at the commit history, this one is a squash and merge (if it gets merged)

jjj333-p · 2024-02-06T06:12:55Z

im fixing it, dont review yet I guess

jjj333-p · 2024-02-06T06:22:56Z

eh im in over my head and not in a mental state where it makes sense to work on this. bit of a waste of a day but 🤷 i got out of bed

jjj333-p · 2024-02-06T09:47:44Z

I've hopefully fixed any potential race conditions, thanks DiamondBurned for the assistance

jjj333-p · 2024-02-13T17:41:57Z

I could be seeing things, but this seems to have made presence with maunium bridges work correctly.

jjj333-p · 2024-02-17T10:33:35Z

one interesting limitation ive discovered is this means presence set through sync doesnt override explicitly set presence through the api.

the only time this matters is for some reason element ios explicitly sets presence to offline when you swipe out of the app to the homescreen. im marking this down as a client bug more than anything because why the hell would it do that

jjj333-p added 8 commits February 5, 2024 12:39

a

a4c4fb9

write out caching logic

18bd660

"a" was never meant to be pushed

5f6dc39

This reverts commit a4c4fb9.

fixed the wrong code

9a307fc

nil maps :woozy:

a6827a3

add offline timeout

fea083b

remove debugging logs

8093f1f

fix recursion hell

de72738

jjj333-p requested a review from a team as a code owner February 6, 2024 05:54

jjj333-p closed this Feb 6, 2024

add mutex locking to reduce race conditions

b9061b9

jjj333-p reopened this Feb 6, 2024

clear up some commenting and whatnot

5b84d46

fix go mod and go sum

93bf55d

jjj333-p added 4 commits February 23, 2024 23:05

Merge branch 'matrix-org:main' into presence-optimization

4872d21

Merge branch 'matrix-org:main' into presence-optimization

b436d43

Merge branch 'matrix-org:main' into presence-optimization

cb76da4

Merge branch 'matrix-org:main' into presence-optimization

00da9b5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing Presence Conflicts #3320

Fixing Presence Conflicts #3320

jjj333-p commented Feb 6, 2024 •

edited

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 13, 2024

jjj333-p commented Feb 17, 2024

Fixing Presence Conflicts #3320

Are you sure you want to change the base?

Fixing Presence Conflicts #3320

Conversation

jjj333-p commented Feb 6, 2024 • edited

Pull Request Checklist

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 6, 2024

jjj333-p commented Feb 13, 2024

jjj333-p commented Feb 17, 2024

jjj333-p commented Feb 6, 2024 •

edited