Teeworlds
IRC / bridge
One-way IRC channel bridge. If you want to be able to send messages to IRC, contact
@Dune
or
@heinrich5991
.
https://www.teeworlds.com/?page=docs&wiki=rules/irc_rules
Between 2020-02-28 00:00:00Z and 2020-02-29 00:00:00Z
[quakenet] Learath2
BOT
2020-02-28 12:46:09Z
is there a tokenizer somewhere in the tw source?
jxsl13
2020-02-28 12:47:49Z
there are tokens being generated.
12:48
oh. now I see what a tokenizer is ._. I'm dumb
Dune
2020-02-28 12:52:17Z
I don't recall seeing one
jxsl13
2020-02-28 12:52:50Z
system.c needs one :'(
Assa
2020-02-28 21:39:22Z
theoretically the splitting of strings by whitespaces is a form of tokenization
21:39
so in python
"I am assa".split(" ")
-> the split funktion is a tokenizer, but a bad one
jxsl13
2020-02-28 22:28:49Z
does the c++ standard library provide such simple functionality?
22:28
22:29
ina simple way
Dune
2020-02-28 22:33:01Z
boost does
jxsl13
2020-02-28 22:45:14Z
oof, external dependencies
22:45
basically defining 20 template parameters to split a string
22:45
boost classic
Dune
2020-02-28 22:46:19Z
just use streams I guess
jxsl13
2020-02-28 22:46:41Z
hm hm
22:46
ss std::stringstream classic
[quakenet] rand
BOT
2020-02-28 22:55:57Z
it's time for strtok
Assa
2020-02-28 23:41:20Z
std::regex
jxsl13
2020-02-28 23:42:34Z
yeah regex for tokenization
Assa
2020-02-28 23:42:47Z
https://stackoverflow.com/questions/16749069/c-split-string-by-regex/16752826
jxsl13
2020-02-28 23:43:37Z
well, if one does not care about performance at all
23:43
use python
23:43
with regex
Assa
2020-02-28 23:44:22Z
no in python I use the tokenizer of sklearn or spacy
jxsl13
2020-02-28 23:44:23Z
to split string
23:44
bette rthan a whole state machine ._.
23:45
finite state machine
Assa
2020-02-28 23:45:23Z
there are dynamic and static tokenizers
23:45
i can give you literature if you want
Exported 30 message(s)
