Use AppendByteOrder to append integers faster

I doubt it'll make a lot of difference for us and we usually have more problems prepending integers, but still it's worth using with Go 1.19: https://pkg.go.dev/encoding/binary#AppendByteOrder