Construction method for the middle part of FIM data

[yysj-zq]

hi there!
How is the middle part of FIM data determined?
Random selection? Is it single or block or mixed in some proportion?

[g0t4]

The StarCoder2 paper references this earlier paper: Efficient Training of Language Models to
Fill in the Middle, find section 4.5 Middle span selection

Instead of trying FIM across syntactic boundaries, such as functions and class bodies, we restrict our ablations to simple, generalizable approaches which are language agnostic.

We select spans in three different ways, splitting randomly by lines, tokens, and characters.

I assume StarCoder2 implemented a similar technique, though I haven't stumbled on explicit confirmation of that...