Products
Resources & Tools
Contact
 
Cart
Please enter a search query.
About
Leskoff
Service &
Support
Home › Resources & Tools › Tokenization Tool

Tokenization Tool

Tokenization is the process of breaking up a string into tokens which usually correspond to words. This is a common task in natural language processing (NLP).

The text to tokenize.
The produced tokens.
Tokenization Tool
Tokenization Tool

Recommended for you

Product
Ea.
Leskoff Møller Scattering Poster
Møller Scattering Poster, English-Labeled
A poster featuring the Møller scattering diagram (Feynman diagram).
$34.99
Leskoff Irish Alphabet Chart
Irish Alphabet Chart, 8.3 x 11.7 in (21.0 x 29.7 cm, Desk Chart, Laminated), English-Labeled
The Irish alphabet chart.
$14.99
Leskoff растение IPA Transcription Poster
растение IPA Transcription Poster
A poster featuring the phonetic transcription of "растение" in the International Phonetic Alphabet (IPA).
$14.99
Leskoff Vanillin Molecule Poster
Vanillin Molecule Poster, Ball-and-Stick Model, Stylized, English-Labeled
A poster featuring the ball-and-stick model (stylized) of the vanillin molecule.
$19.99
Leskoff небосвод Morphemic Analysis Poster
небосвод Morphemic Analysis Poster
A poster featuring the morphemic analysis of the Russian word небосвод.
$14.99

See also

1.
Harnessing the power of the Oxford English Dictionary for linguistic research and NLP applications
How the OED Text Annotator may help bring text mining and natural language processing technologies to the next level.
2.
Chemical elements in Quechua
The list of names of chemical elements in the Quechua language.
3.
Periodic Table Crossword
A daily crossword puzzle for chemical elements.
4.
Ukrainian Keyboard
An on-screen/virtual Ukrainian keyboard for typing Ukrainian text.
5.
The structure and rendering of multipart emails
How do multipart emails work?