SKU # / Keyword
Please enter a search query.
ProductsResources & ToolsServicesAboutContact
ServicesAboutContactTrack order
Track orderView cart
Home › Resources & Tools › Tokenization Tool

Tokenization Tool

Tokenization is the process of breaking up a string into tokens which usually correspond to words. This is a common task in natural language processing (NLP).

The text to tokenize.
The produced tokens.
Tokenization Tool
Tokenization Tool

Recommended for you

Product
Ea.
Leskoff 2027 Wall Calendar
2027 Wall Calendar, Erzya-Labeled, Sunday-Start Layout, Wire-Bound, 11.7 x 8.3 in (29.7 x 21.0 cm)
An Erzya-language wall calendar (2027, Sunday-start format).
$24.99
Leskoff Periodic Table Chart
Periodic Table Chart, 18-Column Layout, Ukrainian-Labeled, 33.1 x 23.4 in (84.1 x 59.4 cm, Poster / Wall Chart)
A Ukrainian-language periodic table of the elements.
$47.59
Leskoff Cardinal Directions Poster
Cardinal Directions Poster, Macedonian-Labeled
A poster featuring the cardinal directions (compass points).
$24.49
Leskoff Monosaccharides Phonetic Wall Chart (Monosaccharides Phonetic Poster)
Monosaccharides Phonetic Wall Chart (Monosaccharides Phonetic Poster), IPA-Labeled (Arabic)
A poster featuring the common monosaccharides.
$19.99
Leskoff хронология IPA Transcription Poster
хронология IPA Transcription Poster
A poster featuring the phonetic transcription of "хронология" in the International Phonetic Alphabet (IPA).
$14.99

See also

1.
Harnessing the power of the Oxford English Dictionary for linguistic research and NLP applications
How the OED Text Annotator may help bring text mining and natural language processing technologies to the next level.
2.
Drugs and medications in Odia
The list of names of common drugs and medications in the Odia language.
3.
Civil engineering terms in Greek
The list of civil engineering terms in the Greek language.
4.
Genomics terms in Odia
The list of genomics terms in the Odia language.
5.
Currencies in Kannada
The list of names of world currencies in the Kannada language.