DDraceNetwork - developer

maybe so but everyone i know who seriously uses rust on linux is linking with mold

True, I'm using mold as well

a7229e9 Use array of size NUM_DUMMIES instead of two variables - Robyt3 b7579b0 Ensure last input send time is reset when disconnecting - Robyt3 178a0da Remove dead code and unused variables - Robyt3 2904c4a Restructure input send checks without ifs - Robyt3 f94053b Rename variable w to Weapon - Robyt3 01be442 Rename variable MDistance to MouseDistance - Robyt3 1fcb8a5 Add GetMinMouseDistance function, remove variable MinDistance - Robyt3 4ec5b04 Use vec2 instead of two separate variables - Robyt3 4f22468 Merge pull request #8443 from Robyt3/Client-Controls-Minor-Refactoring - def-

03:04

def-

[ddnet/ddnet] Pull request closed: #8443 Minor refactoring of CControls

03:04

github-merge-queue[bot]

[ddnet/ddnet] Pull request closed: #8443 Minor refactoring of CControls

ddnet-maps

[ddnet-maps:master] 1 new commit

a123f7e M Essential 2 - ddnet-maps

@trml @Jupstar ✪ do have you got any good ideas for a net message struct that could replace NETMSG_INPUT when sending more than 1 input at once to save size? It seems that nearly 100% of the client's upload bandwidth is already from sending NETMSG_INPUT which makes 8441 pretty expensive in bandwidth. I was able to reduce it from 800 inputs per second maximum to 250 inputs per second maximum by assuming a minimum of 1/4 packets reach the server which gets it to reasonable bandwidth levels but it doesn't perform nearly as well above even 30% packet loss which makes me sad. Currently the client sends about 0.5KB/s-1.75KB/s of NETMSG_INPUT (without dummy), after the PR it's about 6KB-12KB/s. If I limit it to 250 inputs/s it gets down to 2.5KB/s-7KB/s which is more reasonable. The compression ratio on these seems to be around 1.5-1.7:1.0 Other real time games seem to be in the 4-25KB/s range like CSGO/Overwatch/Valorant/Fornite. Those games of course probably already send duplicated inputs, but don't have a way for clients to manually adjust the "prediction margin", it would just be adjusted automatically based on network health so it would be much lower on average. DDNet client also attempts to adjust automatically but seems to fail very bad even in any scenario I tested, even with only 0-50ms random packet delay and 0% packet drop. For the most part there's only 4 pieces of info that change in every individual NETMSG_INPUT which is m_AckGameTick, m_PredTick, m_TargetX, m_TargetY. The other things are limited to your human ability to press a button quickly which is pretty slow. It would be pretty easy to send m_AckGameTick, and m_PredTick only once for the entire message then increment them manually on the server for each contained input but not as easy for m_TargetX, m_TargetY. I have some doubts about how well a naive "binary diff" will work like Juppey suggested, it doesn't seem viable for such tiny sizes. ChatGPT suggested bit packing all the button presses into single bits then sending mouse deltas instead of mouse positions after the initial one, but I'm not sure that helps since the deltas also need to be full ints I think. Also we have m_Fire which is hogging 8 bits minimum.

Currently the client sends about 0.5KB/s-1.75KB/s of NETMSG_INPUT (without dummy), after the PR it's about 6KB-12KB/s. If I limit it to 250 inputs/s it gets down to 2.5KB/s-7KB/s which is more reasonable. The compression ratio on these seems to be around 1.5-1.7:1.0

I'm afraid our servers might get overloaded with that

well that's why I'm writing this message

06:33

alternative would be just set a limit and accept whatever packet loss happens after that

i dont understand why binary patching should not be the solution, isnt the input mostly the same for almost always?

I don't understand how binary patching can define the patch position and the patch data in less bits than bitpacking+huffman (edited)

06:35

people on stackoverflow say it shouldn't be used like this

but the bitpacking and huffman can be applied either way

06:36

actually compression even must be applied

deen

Currently the client sends about 0.5KB/s-1.75KB/s of NETMSG_INPUT (without dummy), after the PR it's about 6KB-12KB/s. If I limit it to 250 inputs/s it gets down to 2.5KB/s-7KB/s which is more reasonable. The compression ratio on these seems to be around 1.5-1.7:1.0

I'm afraid our servers might get overloaded with that

surely there can be some solution though as this input duplication idea was invented by quakeworld in 1996 with like 1% of our current bandwidth (edited)

give me the raw data for an input chain and we can compare which method wins

06:38

i can write the binary patch part very quickly for it

https://stackoverflow.com/questions/48871467/binary-diff-algorithm-for-small-binary-blobs

Binary diff algorithm for small binary blobs

There is a lot of information about binary diff algorithms for pretty big files (1MB+). However, my use case is different. This is why it is not a duplicate. I have a collection of many objects, e...

06:39

idk

well you have nothing to do, i'll do it

wdym

just give me the binary data of your packets

06:40

and i'll try some stuff

forgot about this one: https://github.com/ddnet/ddnet/pull/8423

Replying to egyt literally 10 strings (or 13, if including PRs, who knows), and the trans…

Jupstar ✪

just give me the binary data of your packets

it only sends 1 packet per tick

Tater

it only sends 1 packet per tick

but it sends multiple packet chunks or whatever

06:41

or just give me the 40 bytes of the inputs

06:41

then tell me how tall that packet is on ddnet

06:41

and i try to beat it

it's not that simple

give me the packets before the packer struct is used

ok but what do you mean the packets

06:42

before or after huffman

before

06:42

the raw input data basically, but ofc all 20 or how many u have

ok

then ddnet sends a single packet with all.. this size is interesting bcs that's what to beat (minus the packet header)

why don't you just clone my PR and write it manually then measure the bandwidth used

06:44

you can see outgoing bandwidth even if the server is ignoring your garbage packet data

Tater

@trml @Jupstar ✪ do have you got any good ideas for a net message struct that could replace NETMSG_INPUT when sending more than 1 input at once to save size? It seems that nearly 100% of the client's upload bandwidth is already from sending NETMSG_INPUT which makes 8441 pretty expensive in bandwidth. I was able to reduce it from 800 inputs per second maximum to 250 inputs per second maximum by assuming a minimum of 1/4 packets reach the server which gets it to reasonable bandwidth levels but it doesn't perform nearly as well above even 30% packet loss which makes me sad. Currently the client sends about 0.5KB/s-1.75KB/s of NETMSG_INPUT (without dummy), after the PR it's about 6KB-12KB/s. If I limit it to 250 inputs/s it gets down to 2.5KB/s-7KB/s which is more reasonable. The compression ratio on these seems to be around 1.5-1.7:1.0 Other real time games seem to be in the 4-25KB/s range like CSGO/Overwatch/Valorant/Fornite. Those games of course probably already send duplicated inputs, but don't have a way for clients to manually adjust the "prediction margin", it would just be adjusted automatically based on network health so it would be much lower on average. DDNet client also attempts to adjust automatically but seems to fail very bad even in any scenario I tested, even with only 0-50ms random packet delay and 0% packet drop. For the most part there's only 4 pieces of info that change in every individual NETMSG_INPUT which is m_AckGameTick, m_PredTick, m_TargetX, m_TargetY. The other things are limited to your human ability to press a button quickly which is pretty slow. It would be pretty easy to send m_AckGameTick, and m_PredTick only once for the entire message then increment them manually on the server for each contained input but not as easy for m_TargetX, m_TargetY. I have some doubts about how well a naive "binary diff" will work like Juppey suggested, it doesn't seem viable for such tiny sizes. ChatGPT suggested bit packing all the button presses into single bits then sending mouse deltas instead of mouse positions after the initial one, but I'm not sure that helps since the deltas also need to be full ints I think. Also we have m_Fire which is hogging 8 bits minimum.

but i dunno what u did to measure this

just system process bandwidth usage (edited)

06:45

I used some windows utility program (edited)

06:45

https://www.netlimiter.com/

NetLimiter

NetLimiter, network monitoring and control tool

06:46

NETMSG_INPUT is like 99% of the realtime client upload bandwidth anyway so this is still accurate

morning

1

Tater

@trml @Jupstar ✪ do have you got any good ideas for a net message struct that could replace NETMSG_INPUT when sending more than 1 input at once to save size? It seems that nearly 100% of the client's upload bandwidth is already from sending NETMSG_INPUT which makes 8441 pretty expensive in bandwidth. I was able to reduce it from 800 inputs per second maximum to 250 inputs per second maximum by assuming a minimum of 1/4 packets reach the server which gets it to reasonable bandwidth levels but it doesn't perform nearly as well above even 30% packet loss which makes me sad. Currently the client sends about 0.5KB/s-1.75KB/s of NETMSG_INPUT (without dummy), after the PR it's about 6KB-12KB/s. If I limit it to 250 inputs/s it gets down to 2.5KB/s-7KB/s which is more reasonable. The compression ratio on these seems to be around 1.5-1.7:1.0 Other real time games seem to be in the 4-25KB/s range like CSGO/Overwatch/Valorant/Fornite. Those games of course probably already send duplicated inputs, but don't have a way for clients to manually adjust the "prediction margin", it would just be adjusted automatically based on network health so it would be much lower on average. DDNet client also attempts to adjust automatically but seems to fail very bad even in any scenario I tested, even with only 0-50ms random packet delay and 0% packet drop. For the most part there's only 4 pieces of info that change in every individual NETMSG_INPUT which is m_AckGameTick, m_PredTick, m_TargetX, m_TargetY. The other things are limited to your human ability to press a button quickly which is pretty slow. It would be pretty easy to send m_AckGameTick, and m_PredTick only once for the entire message then increment them manually on the server for each contained input but not as easy for m_TargetX, m_TargetY. I have some doubts about how well a naive "binary diff" will work like Juppey suggested, it doesn't seem viable for such tiny sizes. ChatGPT suggested bit packing all the button presses into single bits then sending mouse deltas instead of mouse positions after the initial one, but I'm not sure that helps since the deltas also need to be full ints I think. Also we have m_Fire which is hogging 8 bits minimum.

What issue does this 12x increase in traffic solve?

Tater

surely there can be some solution though as this input duplication idea was invented by quakeworld in 1996 with like 1% of our current bandwidth (edited)

Sure, more expensive servers

deen

Sure, more expensive servers

Or less players per server

Learath2

What issue does this 12x increase in traffic solve?

view at timestamp https://www.youtube.com/watch?v=W3aieHjyNvw&t=1891s

GDC

Overwatch Gameplay Architecture and Netcode

06:59

There is a demo here: https://github.com/ddnet/ddnet/pull/8441

Allow client to re-send past inputs to the server by sjrc6 · Pull R...

Requires #8432 Does 2 main things. If the client has enough prediction margin that it can send send duplicated inputs before the original input reaches the "simulation tick" on the serve...

Isn't this dangerous? It means we trust the client more now and the client can manipulate the past? I'm wondering if this can give attackers an advantage to cheat

no

07:00

client cannot change inputs that it already sent to server after https://github.com/ddnet/ddnet/pull/8432

Refactor Server Input Handling by sjrc6 · Pull Request #8432 · ddne...

The motivation for this PR is to make the server side input code easier to understand, and improve the way the input buffer for each client is handled. First change: This PR should allow clients to...

07:00

no extra trust

deen

Isn't this dangerous? It means we trust the client more now and the client can manipulate the past? I'm wondering if this can give attackers an advantage to cheat

even if, it can only manipulate queued inputs.. so there isn't really an advantage that isnt there yet.. it could not change past inputs (in the sense that the logic already processed them)

it has danger after #8372

https://github.com/ddnet/ddnet/pull/8372

Presend inputs by JSaurusRex · Pull Request #8372 · ddnet/ddnet

The goal of this pr is to improve antiping player prediction 2024-05-17.12-55-33.mp4 This change sends inputs that already arrived at the server to the clients ...

but ddnet is not PvP so really it doesn't matter

07:02

the client could already keep a local buffer of inputs if it really wanted

07:02

you could play the entire map offline then play it back as a replay bot

Tater

it has danger after #8372

but even here, if one player does not use prediction margins, the other player doesnt really gain anything

in PvP you could do something like trick your opponent into thinking you're in 1 position then you change your inputs on the server so you're actually in another position

07:03

CS:GO had this vunerability and it is in cheats (edited)

07:04

because they let the client modify inputs it already sent to the server buffer

07:04

for some reason

Tater

in PvP you could do something like trick your opponent into thinking you're in 1 position then you change your inputs on the server so you're actually in another position

but still.. this only works if u use a reasonable high inputs queue in first place

07:05

a good client doesn't need a high prediction margin

a cheating client doesn't care

You can use less bytes for targetx and targety if their deltas are bounded. Similarly for ackgametick and predtick, you can only send deltas which are bounded

Tater

a cheating client doesn't care

so the cheating client sets a high prediction margin with low latency network basically? mh yeah that could work

bounding the delta of targetx targety would be a change in client capabilities, is that ok?

07:06

idk how you even enforce that

Tater

bounding the delta of targetx targety would be a change in client capabilities, is that ok?

Is it even physically possible to go faster than like 2 bytes at a time

Tater

but ddnet is not PvP so really it doesn't matter

sometimes it is

Ewan

sometimes it is

t0 player has spoken

Learath2

Is it even physically possible to go faster than like 2 bytes at a time

targetx targety is scaled by zoom so you can move abritrarily fast in the client already

Tater

but ddnet is not PvP so really it doesn't matter

hmm, what about pvp mod hosters?

Jupstar ✪

t0 player has spoken

not often

Tater

targetx targety is scaled by zoom so you can move abritrarily fast in the client already

Ah, that's a shame

Nouaa

hmm, what about pvp mod hosters?

Nouaa

hmm, what about pvp mod hosters?

they will not receive this packet

07:08

ddnet only

Perhaps a magic delta value that indicates the next targetx and targety are not delta but full

07:08

0xFFFF

Nouaa

hmm, what about pvp mod hosters?

also as I said there is no risk because the client cannot change values it already sent

I see

The rest of the input packet should compress very well though, do you do that yet?

the compression performance of all packets is rather poor because the huffman table is 15 years old I think

07:10

I only got like 1.5:1

I would be quite surprised if that is an actual issue

idk

07:11

perhaps not

07:11

1.5:1 is not great either way

But why rely on huffman anyway when we know the pattern already, the most common thing that will happen is that the input won't change. That's 1 bit you need to sneak in somewhere

yes that's true, we can do better than huffman

Why was huffman chosen in the first place? I think the nature of this kind of data makes it hard to compress

huffman is just bonus

Ewan

Why was huffman chosen in the first place? I think the nature of this kind of data makes it hard to compress

ask magnus

07:13

although I don't think it's a terrible choice

07:14

it does better than nothing

Tater

huffman is just bonus

Exactly anything our naive but effective bit cant help with. Huffman can get a bit better

there are instances where manually minimizing your data actually increases the overall size after compression because it has more entropy at the compressor, but I doubt this will be an issue for us because our huffman is very weak.

07:16

turning duplicate bits into smaller non-duplicate bits makes compression worse

07:16

sometimes

Ewan

Why was huffman chosen in the first place? I think the nature of this kind of data makes it hard to compress

Wherever structured data is involved, huffman coding usually works well-ish. It is also rather performant compared to alternatives

07:16

What would you use?

Tater

there are instances where manually minimizing your data actually increases the overall size after compression because it has more entropy at the compressor, but I doubt this will be an issue for us because our huffman is very weak.

Can I get a short TL;DR why we are talking about minimizing data?

we would like to send duplicated input packets to minimize the lag that happens when packet loss occurs, but the total bandwidth is already 99% input packets so its important we minimize their size (edited)

Avolicious

Can I get a short TL;DR why we are talking about minimizing data?

He wants to resend inputs that havent been acknowledged by the server yet

Oh I see, have you tried it already?

07:19

Or is everything theoretical?

demo in here https://github.com/ddnet/ddnet/pull/8441

07:19

possibly this would also make DDoS less effective if we can make the inputs small enough

Only a meager 12x increase in traffic

Tater

possibly this would also make DDoS less effective if we can make the inputs small enough

i doubt that, but it makes it less horrible for smaller lags

Learath2

Only a meager 12x increase in traffic

Damn

12x is worst case if client is spamming their keyboard

also it assumes you actively use prediction margins

Jupstar ✪

also it assumes you actively use prediction margins

in theory the client should automatically adjust prediciton margin based on packet loss but this is a future issue

Tater

in theory the client should automatically adjust prediciton margin based on packet loss but this is a future issue

yes but then the ddos pattern simply changes to 2 second massive attacks

07:21

and then it doesnt work either way (edited)

Tater

possibly this would also make DDoS less effective if we can make the inputs small enough

Or worse because now our clients are also effectively ddosing us with inputs thinkies

I doubt it

How big is the rest of the input packet btw?

wdym rest

Except the 4 fields we already discussed

40bytes for input + the 12 bytes for 3 extra info ints

What are in the extra info ints? Also slow changing stuff?

m_AckGameTick, m_PredTick, m_Size

07:23

m_Size is actually fixed at 40 and could maybe be deleted?

07:24

its only incase we increase the size of the input struct

07:24

I assume?

Idk why it exists, been years since I last looked into input handling

much of it is bad so it would not surprise me if we can remove it

07:25

the client and server sent the input struct as abstract data which gets copied back into the actual struct later

07:25

also I think the chance of us changing the input struct is pretty low (edited)

If we chunk the input packet into bytes (or twobytes). We can have two (or one) byte(s) whose bits can track whether their respective bytes (or twobytes) have changed. That would compress 40 bytes to 2-1 bytes in the common case that input doesn't change

I would be interested in game tick negotiation, so we could run 100 tick servers instead of 50

Learath2

Or worse because now our clients are also effectively ddosing us with inputs thinkies

if DDoS is killing server performance then duplicate inputs would not help/make it worse, but if the DDoS is clogging network bandwidth then it would help a lot

but i guess this will kill the physics, no? they are heavily coupled to the ticks

Avolicious

I would be interested in game tick negotiation, so we could run 100 tick servers instead of 50

this changes physics too much I think

07:27

there was discussion and several prototypes on github

Tater

this changes physics too much I think

Absolutely

it makes gores way easier :)

Yeah 100ticks is a no go I think

You could have a way to allow it for new maps, but it might feel strange for players used to 50 ticks

Avolicious

I would be interested in game tick negotiation, so we could run 100 tick servers instead of 50

for gores it could work

07:29

since it's not as bound to weird physics hacks

Anyone who implements it better have very concrete proof that if set at 50hz it still behaves the exact same as it used to

Avolicious

I would be interested in game tick negotiation, so we could run 100 tick servers instead of 50

we use sv_high_bandwidth on blockworlds, it might slightly change the physics but cosmetics are way smoother when it's enabled (edited)

it does not change physics, just increases snapshot rate to clients

07:30

25->50 snaps/s

Nouaa

we use sv_high_bandwidth on blockworlds, it might slightly change the physics but cosmetics are way smoother when it's enabled (edited)

Interesting

When neural network based prediction?

would be nice to have on offical servers

Jupstar ✪

since it's not as bound to weird physics hacks

Mhmmm, never tried before, but it could

Tater

would be nice to have on offical servers

We would have to check and see how much more bandwidth it would use. Snapshots are delta compressed so it shouldnt be too too bad

But I'd be in favor of negotiation in first place

Not just setting the server to 100 ticks

I assume its near double

Avolicious

Interesting

so instead of 50 ticks/sec it's 100 ticks/sec

Tater

I assume its near double

I would guess about 1.5x

Nouaa

so instead of 50 ticks/sec it's 100 ticks/sec

NO

07:32

The state is snapshot more often. The state doesn't mutate more often. The tickrate refers to how quickly the state mutates

1

Learath2

We would have to check and see how much more bandwidth it would use. Snapshots are delta compressed so it shouldnt be too too bad

it depends, just because they are delta compressed doesn't mean it won't double it. Only if full snapshots stay the same and delta snapshots increase would it have less than 2x increase (edited)

07:33

but if most of the bandwidth is already deltas then it will 2x

07:33

since you can't get smaller than that

Yeah would have to observe

Tater

it does not change physics, just increases snapshot rate to clients

oh okay

also it affects demo size

07:34

on client PCs

Tater

m_Size is actually fixed at 40 and could maybe be deleted?

so all input packets are always the same size? can i assume that?

yes

nice

07:34

that makes it even easier

Nouaa

oh okay

It appears more smooth because you get a snapshot about every tick. Instead of the default every other tick

Learath2

The state is snapshot more often. The state doesn't mutate more often. The tickrate refers to how quickly the state mutates

I guess this is true

07:35

idk how much that helps but it does

07:35

if sending more snaps means you send less deltas per tick it could be lower (edited)

I was referring to Noa's message still

ah right

07:36

unintentially interesting idea tho xd

Anyway, idk. I don't think we can guesstimate much since I actually don't remember how much of the traffic is deltas, but the sent deltas may or may not get smaller depending on whether we have things that change slower than 50hz faster than 25hz

I think it should also reduce average latency by 20ms right?

It should reduce it but I don't know why it would be by a flat number

well on average

07:40

at any given point in time there is between 0-40ms until the next snap the server sends you on 25tps

07:40

at 50tps it's 0-20ms until the next one

Ah, I see what you mean. Yeah that sounds about reasonable

Increasing the speed of sending inputs would probably increase the accuracy of the antibot (at least my implementation), as often the change of inputs looks extremely strange, but in fact the reason for this is packet delays

you're using client side antibot?

server side ofc

the server already recieves the full 50 inputs per second for every player. There can't be more than that unless you change the physics

07:44

only clients are limited to 25hz (for information about other players) (edited)

This is in the perfect case, but in reality there are delays

Tater

the server already recieves the full 50 inputs per second for every player. There can't be more than that unless you change the physics

I guess sending it quicker wouldn't technically change the physics per se, but the feel

I'm not sure what you mean

07:45

client side prediction ensures the feel is the same regardless of how fast your input reaches the server

@Tater so i have the binary diff now.. if u can now tell me how i can manually call compress from cpp code i can check how much better binary patching performs if i use ddnet's huffman compression

Faulty

This is in the perfect case, but in reality there are delays

@Tater 's new patch would mean input packets would be less likely to be lost. Which would make your antibot get a more continuous appearing data stream. But you should probably be correcting for random transmission delays as those convey no information really

Jupstar ✪

@Tater so i have the binary diff now.. if u can now tell me how i can manually call compress from cpp code i can check how much better binary patching performs if i use ddnet's huffman compression

I don't have some sort of compression testing framework for you to easily call a single function on the input data. You have to pack all the data for each input message (the 3 ints + player_input) into a single segment of memory, then add each segment of it to the message as an int.

ok i'll just create a fake packet using the packer thing

07:48

lets see if that works

yeah exactly

07:48

the packet can be garbage as far as the server knows

Jupstar ✪

ok i'll just create a fake packet using the packer thing

you just need to make sure that whatever you put into the message can theoretically be converted back into the full data on the other side with all the information you give it

Learath2

@Tater 's new patch would mean input packets would be less likely to be lost. Which would make your antibot get a more continuous appearing data stream. But you should probably be correcting for random transmission delays as those convey no information really

You right, I meant continuity exactly, delays could be handled but missed data not

this guy on reddit had an interesting idea

Tater

you just need to make sure that whatever you put into the message can theoretically be converted back into the full data on the other side with all the information you give it

yeah since the input packets are same size that's super ez xd

if you turn all the data into 0s then the generic compressor should make it really small

There are many ways to compress slowly changing data

less ways when it's only 52 bytes

Well you can do exactly what he says and it would work fine, but I wouldn't really use something like lz4. Just rle

07:55

Or you can do what I suggested with one or two bytes tracking the rest of the bytes

I'm still a little dissapointed with the 1.5:1 by the huffman. like 80% of the input struct should always be zeros without any special tricks (edited)

so how can i identify my packet in wireshark now lmao

Jupstar ✪

so how can i identify my packet in wireshark now lmao

wireshark is hard sorry lol

07:59

look up some tutorials

chiller would know it

true

Tater

I'm still a little dissapointed with the 1.5:1 by the huffman. like 80% of the input struct should always be zeros without any special tricks (edited)

Well the table is not for just inputs (I don't even know what the table is generated by/for). Huffman doesn't perform that well without seeing a lot of input

I don't think the size of data to be comrpessed matters if the huffman table is fixed? (edited)

We could have a huffman tree just for inputs and that might do better. It's have lots of sequences of zeros on short codes

@Tater i think the constant 136 bytes are my packet

08:01

and the bigger ones are the normal resent packets

I do not know how to use wireshark idk what you want from me lol

08:01

136bytes for 15 inputs?

13 inputs

ok

08:01

quite good I think right?

i guess so

hmm

using brotli compression i get exactly the same lol

actually that's 6.8KB/s

08:02

I think

08:03

which is still better than what I had at 13 inputs probably

Jupstar ✪

using brotli compression i get exactly the same lol

ah but it would loose if i'd add packet headers rip

are you spinning your mouse in circles?

08:03

that matters a lot

Tater

are you spinning your mouse in circles?

yes

ok

i tried to jump and spin

08:03

and move

08:03

and press alt + f4 at the same time

08:03

lmao

lol

Jupstar ✪

and press alt + f4 at the same time

Rq

Jupstar ✪

using brotli compression i get exactly the same lol

But how fast is yours vs brotli?

Learath2

But how fast is yours vs brotli?

the brotli part is only interesting bcs i wrote it in rust

08:04

was just a test

A lot of the big math smart guy optimizations don't kick in until sizes much larger than a couple hundred bytes

Learath2

A lot of the big math smart guy optimizations don't kick in until sizes much larger than a couple hundred bytes

that's correct

08:05

but i think for a general purpose compression that's still pretty good

Yeah I think that's pretty good. How did you end up implementing it?

also brotli is not designed for binary data actually lmao

136 bytes is about 5:1 which is the same as the amount of zeros to data in the orignal struct that I calculated a few minutes ago

Learath2

Yeah I think that's pretty good. How did you end up implementing it?

the binary patch?

Yes

very straight forward actually

let mut writer = Vec::new();
        let empty = vec![
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        ];
        let mut cur_last = &empty;
        for a in a_bytearr.iter() {
            fn diff(old: &[u8], new: &[u8], writer: &mut Vec<u8>) {
                for (index, old) in old.iter().enumerate() {
                    writer.push(new[index].wrapping_sub(*old));
                }
            }
            diff(cur_last, a, &mut writer);
            cur_last = a;
        }

08:07

a_bytearr is simply the i32 converted to u8

08:07

so for all input packets i simply create the binary diff towards the previous one

I think I need my coffee and breakfast now. That looks like it encodes 40 bytes to 40 bytes diff

Learath2

I think I need my coffee and breakfast now. That looks like it encodes 40 bytes to 40 bytes diff

it simply takes the difference

08:09

thus creating lots of 0s

08:09

that's the whole point of it

08:10

and if not 0s then at least some smaller ints

Ah, so you just let huffman coding take care of it

yeah compression is applied nonethenless

08:11

i basically just want to help the compressor a bit

08:11

since inputs are always same size and always similar values

I guess the longer sections of 0 are easier on the huffman table we have for some reason

couldn't you just xor the previous input with the next one?

08:11

like the guy on reddit said

You can

well nice

08:12

but is xor even better? well i dunno

08:12

my math brain is off rn

I can't imagine how much it changes rn. Not before breakfast

is this not equivalent to xor?

08:13

if both are 40bytes the difference is actually just the xor?

08:13

idk what your code does

Tater

idk what your code does

Bytewise sub

hmm

08:13

bitwise is better tho?

08:14

or it needs bytewise?

Tater

bitwise is better tho?

maybe int wise is even better than byte wise

There is no bitwise sub :F

since it doesnt destroy the whole int

But bitwise xor of the entire thing is probably better than everything

yes thats what I was thinking

but anyway

08:15

the idea is clear

08:15

i'd try smth like that

ok but 6.8KB/s is not acceptable yet I think

no matter how good ur compression is.. context based information is just OP (edited)

that's still 12x increase?

Well you'll need to set up a better way to test this and test a couple things

Tater

that's still 12x increase?

really? wow, didnt know ddnet is so efficient upload wise

the input is the ONLY upload

08:16

lol

@Tater but even if u only pack the latest 5 inputs with a similar size it would already help

08:16

15 inputs is simply a lot

Try the xor thing, try the delta compression I suggested, try xor followed by run length encoding

you don't get the same packet loss protection characteristics with a limit like that (edited)

08:17

the overwatch guy said they send 15 inputs, he was also like "it compresses super well" without explaining how they do it xd

If you are ambitious, try a new huffman table

how do u even restore from xor?

08:17

i dont understand anything

08:17

ah right

08:17

u know one side

08:17

lmao

Jupstar ✪

how do u even restore from xor?

Xor

ok i do xor

08:18

just for u

I wonder if bitpacking+xor would be better

08:19

less zeros is less data

brotli already gets worse with xor xD

Actually one issue, does this not make the packet loss tolerance 0? If somehow an input packet truly gets lost, how would you recover?

now let me move it to cpp

Learath2

Actually one issue, does this not make the packet loss tolerance 0? If somehow an input packet truly gets lost, how would you recover?

all the inputs are in the same packet

08:20

oh

Tater

all the inputs are in the same packet

And the window of inputs is always from the last acknowledged input seq? (Idk if we even keep track of that)

no we won't do that

08:20

just the size of prediction margin buffer

91 bytes

:O

08:20

that's significant

so xor wins for huffman

is this with the 3 extra info ints as well?

08:21

or just 40 bytes of input

Tater

is this with the 3 extra info ints as well?

thats the inner size of the UDP packet

Tater

no we won't do that

So if somehow a sequenced input gets lost from the head before it's processed by the server, how does the client recover? Can it recover?

the inputs are only sequenced within a single packet (edited)

08:22

each packet can be processed without any others

08:22

you send redundent inputs incase one of the packets gets dropped the others have the same data, so the server can fill it in

08:22

that's the whole idea

Say we have a window of 3 inputs for simplicity. We have a, b, c. We send a packet with a, b, c. The nex packet will be b, c, d. If the first is lost. a is truly lost, no?

meloƞ

can someone try and reproduce crashing the client by rendering a demo, setting the playback time to 0.1 - forwarding a bit and then quitting out of it without finishing rendering using the X on the bottom right? - my client became unresponsive for a solid minute, outputted me this:

2024-06-04 00:18:38 I demo_player: Stopped playback
2024-06-04 00:18:38 I demo_player: Loading demo 'demos/replays/asdasd.demo'
2024-06-04 00:18:38 I datafile: could not open 'maps/Adrenaline 3.map'
2024-06-04 00:18:38 I videorecorder: Recording to 'videos/asdasd.mp4.mp4'
2024-06-04 00:18:59 I client: disconnecting. reason='unknown'
2024-06-04 00:18:59 I video_recorder: ------------
2024-06-04 00:18:59 I video_recorder: ------------
2024-06-04 00:18:59 I demo_player: Stopped playback

and then closed (using DDNet 18.2) this is the output on latest upstream:

2024-06-04 00:22:47 W videorecorder/libav: Too many bits 32768.000000 > 12288 per frame requested, clamping to max

2024-06-04 00:22:47 I videorecorder: Recording to 'videos/aseasdasdasd.mp4'

2024-06-04 00:22:51 I videorecorder/libav: frame I:114   Avg QP:14.91  size:177571

2024-06-04 00:22:51 I videorecorder/libav: frame P:412   Avg QP:24.00  size: 10011

2024-06-04 00:22:51 I videorecorder/libav: frame B:831   Avg QP:25.97  size:  3511

2024-06-04 00:22:51 I videorecorder/libav: consecutive B-frames: 12.7%  9.6% 22.3% 55.4%

2024-06-04 00:22:51 I videorecorder/libav: mb I  I16..4: 32.8% 25.2% 41.9%

2024-06-04 00:22:51 I videorecorder/libav: mb P  I16..4:  0.7%  0.4%  0.5%  P16..4:  7.7%  2.8%  2.0%  0.0%  0.0%    skip:85.9%

2024-06-04 00:22:51 I videorecorder/libav: mb B  I16..4:  0.3%  0.0%  0.0%  B16..8:  7.4%  1.6%  0.5%  direct: 1.3%  skip:88.7%  L0:51.0% L1:37.3% BI:11.7%

2024-06-04 00:22:51 I videorecorder/libav: 8x8 transform intra:24.9% inter:51.0%

2024-06-04 00:22:51 I videorecorder/libav: coded y,uvDC,uvAC intra: 30.4% 34.5% 30.3% inter: 2.5% 1.3% 0.7%

2024-06-04 00:22:51 I videorecorder/libav: i16 v,h,dc,p: 77% 21%  2%  0%

2024-06-04 00:22:51 I videorecorder/libav: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 72%  4% 24%  0%  0%  0%  0%  0%  0%

2024-06-04 00:22:51 I videorecorder/libav: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 41% 30%  7%  3%  3%  4%  3%  4%  4%

2024-06-04 00:22:51 I videorecorder/libav: i8c dc,h,v,p: 68% 18% 12%  2%

2024-06-04 00:22:51 I videorecorder/libav: Weighted P-Frames: Y:0.0% UV:0.0%

2024-06-04 00:22:51 I videorecorder/libav: ref P L0: 60.3%  2.7%  9.1% 27.8%

2024-06-04 00:22:51 I videorecorder/libav: ref B L0: 73.7% 21.5%  4.8%

2024-06-04 00:22:51 I videorecorder/libav: ref B L1: 92.7%  7.3%

2024-06-04 00:22:51 I videorecorder/libav: kb/s:9651.27

2024-06-04 00:22:51 I videorecorder/libav: Qavg: 65418.816

2024-06-04 00:22:51 I demo_player: Stopped playback

Several demo rendering related crashes were fixed in nightly so it's likely this was already fixed and only happens in 18.2 and earlier if you can't reproduce it anymore

Then applying the deltas, you can no longer get to a consistent state ever

Robyt3

Several demo rendering related crashes were fixed in nightly so it's likely this was already fixed and only happens in 18.2 and earlier if you can't reproduce it anymore

i was able to not reproduce the same behaviour, but i still got the client to freeze with latest upstream

I don't understand what your concern is

I think before we apply a resending patch like https://github.com/ddnet/ddnet/pull/8441, we should first figure out if packet loss like that is even a thing on ddnet servers

08:24

it adds quite the complexity and as such should only be applied if it actually does something

08:24

not just theoretically

Previously since the input packet contained the entire thing, even if a couple were lost, you could always resync because well the entire thing is there

I think it happens of course

meloƞ

i was able to not reproduce the same behaviour, but i still got the client to freeze with latest upstream

Please open an issue for that

will do

heinrich5991

I think before we apply a resending patch like https://github.com/ddnet/ddnet/pull/8441, we should first figure out if packet loss like that is even a thing on ddnet servers

ask any gores player and they will tell you how many hundreds of times they die due to small lag

yeah was about to say

08:25

lets join KoG Turkey xd

then I guess we should do the measurement on KoG turkey

it's silly to say packet loss does not happen

08:26

even if 1/10000 packets is dropped we have zero redundancy

08:26

so it will always cause client desync

why? AFAIK packet loss is rare on the actual internet

packet delay is effectively the same

Tater

it's silly to say packet loss does not happen

but tbf, then it's about if u want to permanentely play with prediction margin Because otherwise it could never recover from it anyway

08:26

even with automatic prediction margin not

08:27

it cannot guess random laggs

08:27

*drops

I already mentioned it should adjust automatically

but even then

all these ideas are well documented and impliemented in multiplayer games for like 20 years

how should it know the 10000th packet will drop

I am not creating any ideas here

that doesnt matter xD

08:27

it simply makes no sense

wdym

08:27

I mean 1/10000 chance

08:27

not every 1/10000 packets sequentailly

Tater

I don't understand what your concern is

This method does not guarantee a sequenced input will be never lost, unless it's coupled with server side acks and a window that grows infinitely. If you let the window slide, then you should make sure the entire packet is recoverable without needing the previous packet ever

if u have a prediction margin of 1ms and the 10000th packet will drop, how do u want to handle that?

08:28

how does the automatic prediction timer prepare for that?

if you have a margin of 1ms you cannot do anything

see, so you either have to accept to play with at least 20ms ALWAYS

please watch the GDC talk I can't explain it as well as them

or it doesnt really work anyway

Tater

please watch the GDC talk I can't explain it as well as them

but any kind of prediction margin always also means more input latency, which is totally ok for ddrace

Jupstar ✪

see, so you either have to accept to play with at least 20ms ALWAYS

Is that not what you already do? The client is always ahead of the server

the margin should increase automatically based on the server not acking your input when you expect

i just want it to be clear

08:29

without prediction margin of 20 or actually even 30 it's basically useless

08:29

without mininmum*

Jupstar ✪

but any kind of prediction margin always also means more input latency, which is totally ok for ddrace

view https://github.com/ddnet/ddnet/pull/8372

Tater

the margin should increase automatically based on the server not acking your input when you expect

yes sure, but not if it's only every 10000th packet

08:30

u can never guess that xD

08:30

so a minimum prediction margin would be around 30ms

sure

Learath2

Is that not what you already do? The client is always ahead of the server

yeah but only a bit

Jupstar ✪

yeah but only a bit

I'd guess it's not less than 20ms

BlaiZephyr

[ddnet/ddnet] Issue opened: #8445 Client Becomes Unresponsive During Demo / Video Rendering

steps to reproduce: select a demo render cut to video and set the playback time to 0.1 pause the demo rendering forward a bit Close the rendering happens on steam stable, steam nightly and latest upstream this is the console output on latest and nightly steam: ``` 2024-06-04 10:28:29 I demo_player: Stopped playback 2024-06-04 10:28:29 I demo_player: Loading demo 'demos/auto/fddfdssdgsd.demo' 2024-06-04 10:28:29 I datafile: could not open 'maps/Samsara VI.map' 2024-06-04 10...

if you start dropping inputs don't you think it's better that you gain a little latency than you are no longer able to controll your tee?

Learath2

I'd guess it's not less than 20ms

it's 10

08:31

by default

Tater

if you start dropping inputs don't you think it's better that you gain a little latency than you are no longer able to controll your tee?

i'd say it's at least fair to say it's a change that is noticable

Tater

if you start dropping inputs don't you think it's better that you gain a little latency than you are no longer able to controll your tee?

I'd really like to see some data on this. if it is not actually happening like you say, then this patch doesn't do anything

bcs antiping will "lag" more

Huh, I remembered the default being 20

heinrich5991

I'd really like to see some data on this. if it is not actually happening like you say, then this patch doesn't do anything

the worlds best gores player could not solo a map on CHN servers

08:33

you just have to press CTRL+SHIFT+D then CTRL+SHIFT+G and the ingame graph will show you your packet delays

I guess the PR is not about improving european player's experience on chinese servers, is it? but yea, I agree that there is packet loss to china

08:34

china has a very weird connection to the rest of the internet though, with the great firewall interfering and all

what exactly do you want me to do to collect data

08:35

will you say my ISP is specially bad compared to average

write a small patch for the ddnet server that checks what inputs were missed

the server can't know

Dont we have sequence numbers on inputs?

write a small patch for the ddnet server and client to see what inputs were missed

Learath2

Dont we have sequence numbers on inputs?

client only sends input sometimes

08:36

it assumes the server will reuse past input

08:36

which btw makes packet drops worse

heinrich5991

write a small patch for the ddnet server and client to see what inputs were missed

the client already has the graph that shows you network issues

08:37

just go join some servers and view it

which of these graphs? they look pretty stable to me

it's too early anyway

the gametime graph shows me when I tab into ddnet

4 p.m. and turkey servers suck xddd

It might be more useful during ddos too

ddos is too heavy

08:40

i doubt this patch really helps

Jupstar ✪

4 p.m. and turkey servers suck xddd

but why? if it's not bandwidth limited, why should packets be dropped?

08:41

(and if it were bandwidth limited, this patch would make it worse)

heinrich5991

but why? if it's not bandwidth limited, why should packets be dropped?

i dunno, i guess the bandwidth is kinda limited, whole europe then sends packets

heinrich5991

but why? if it's not bandwidth limited, why should packets be dropped?

Turkey is connected to the backbone using fishing wire

i mean it's probably about short spikes and rerouting etc.

why would ddnet be immune to packet loss but other games routing over UDP are not?

do you have something where I can read about it? I'm not into watching long videos (edited)

"The best servers in whole Europe are the USA ones" - Jupstar

heinrich5991

do you have something where I can read about it? I'm not into watching long videos (edited)

do you want evidence of packet loss or explanation of my PR

Tbf there might not even be much written about it since it's such an obvious thing to do. They basically treat inputs as semi-vital

people discuss sending duplicated input pretty often https://www.reddit.com/r/gamedev/comments/2f14s3/handling_packet_loss_in_a_clientserver_network/ck549s8/

nonotan's comment on "Handling packet loss in a client-server netwo...

Explore this conversation and more from the gamedev community

Tater

do you want evidence of packet loss or explanation of my PR

packet loss

@archimede67 Control find (in the in-game console) seems to highlight more then it should when searching for strings with cyrillic characters (edited)

byte vs "char" count, probably?

heinrich5991

packet loss

Is this not very tough to show? All teeworlds servers get some amount of ddos, how would you rule that out? My internet here is on some days shit, how would you rule that out?

Learath2

Is this not very tough to show? All teeworlds servers get some amount of ddos, how would you rule that out? My internet here is on some days shit, how would you rule that out?

by writing a small patch on the server, we can see how it affects actual players

I mean if packet loss was extremely rare, tcp wouldn't really need to exist, nor would we need vital messages

packet loss is rare AFAIK