Please note, this is a STATIC archive of website www.w3resource.com from 19 Jul 2022, cach3.com does not collect or store any user information, there is no "phishing" involved.
w3resource

NLTK Tokenize: Exercises with Solution

Python NLTK Tokenize [9 exercises with solution]

What is Tokenize?

Tokenization is the process of demarcating and possibly classifying sections of a string of input characters. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input.

1. Write a Python NLTK program to split the text sentence/paragraph into a list of words.
Click me to see the sample solution

2. Write a Python NLTK program to tokenize sentences in languages other than English.
Click me to see the sample solution

3. Write a Python NLTK program to create a list of words from a given string.
Click me to see the sample solution

4. Write a Python NLTK program to split all punctuation into separate tokens.
Click me to see the sample solution

5. Write a Python NLTK program to tokenize words, sentence wise.
Click me to see the sample solution

6. Write a Python NLTK program to tokenize a twitter text.
Click me to see the sample solution

7. Write a Python NLTK program to remove Twitter username handles from a given twitter text.
Click me to see the sample solution

8. Write a Python NLTK program that will read a given text through each line and look for sentences. Print each sentence and divide two sentences with "==============".
Click me to see the sample solution

9. Write a Python NLTK program to find parenthesized expressions in a given string and divides the string into a sequence of substrings.
Click me to see the sample solution

 

More to Come !

Do not submit any solution of the above exercises at here, if you want to contribute go to the appropriate exercise page.

[ Want to contribute to Python - NLTK exercises? Send your code (attached with a .zip file) to us at w3resource[at]yahoo[dot]com. Please avoid copyrighted materials.]



Python: Tips of the Day

Find current directory and file's directory:

To get the full path to the directory a Python file is contained in, write this in that file:

import os 
dir_path = os.path.dirname(os.path.realpath(__file__))

(Note that the incantation above won't work if you've already used os.chdir() to change your current working directory, since the value of the __file__ constant is relative to the current working directory and is not changed by an os.chdir() call.)

To get the current working directory use

import os
cwd = os.getcwd()

Documentation references for the modules, constants and functions used above:

  • The os and os.path modules.
  • The __file__ constant
  • os.path.realpath(path) (returns "the canonical path of the specified filename, eliminating any symbolic links encountered in the path")
  • os.path.dirname(path) (returns "the directory name of pathname path")
  • os.getcwd() (returns "a string representing the current working directory")
  • os.chdir(path) ("change the current working directory to path")

Ref: https://bit.ly/3fy0R6m