This paper describes our approach (UR-
mSBD) to address the shared task on Sentence
End and Punctuation Prediction in NLG Text
(SEPP-NLG) organised as part of SwissText
2021. We participated in Subtask 1 (fully un-
punctuated sentences – full stop detection) and
submitted a run for every featured language
(English, German, French, Italian). Our sub-
missions are based on pre-trained BERT mod-
els that have been fine-tuned to the task at hand.
We had recently demonstrated, that such an ap-
proach achieves state-of-the-art performance
when identifying end-of-sentence markers on
automatically transcribed texts. The difference
to that work is that here we use language-
specific BERT models for each featured lan-
guage. By framing the problem as a binary
tagging task using the outlined architecture we
are able to achieve competitive results on the
official test set across all languages, with Re-
call, Precision, F1 ranging between 0.91 and
0.96 which makes us joint winners for Recall
in two of the languages. The official baselines
are beaten by large margins.