Data Mining Project 1: Text MiningThe primary purpose of this assignment is to get familiar with text preprocessing, feature extraction, feature selection, classification, and clustering. You can work individually or in a two-person team. Setup: You will use Python for this project and need the NLTK toolkit and the scikit-learn library. Python 3 is recommended. To install them, you should install pip3 first. Then, run the following command: II pip3 install -U sklearn nitk Download the mini 20 newsgroups dataset. The documents are distributed into 20 directories, corresponding to the 20 news groups. Each file in the directory corresponds to one document. For plotting curves, you may consider matplotlib, as shown in this example. More examples can be found on the sklearn or other websites. IFdp3 install -U matplotlibPart 1: Text preprocessing and feature extractionIn this task, you will develop the program feature-extract.py, which has the following usage.IIpython feature-extract.py directory_of_newsgroups_data feature_definition_file class_definitiowhere input: directory_of_new
sgroups_data is the directory of the unzipped newsgroups data, output: feature_definition_file contains (term, feature_id) pairs, output: class_definition_file contains (class_name, class_id) pairs, output: training_data_file has a specific format, which will be described later.feature-extract.py contains the following two components 1.1 and 1.21.1 document preprocessing. For each document, the preprocessing steps include split a document to a list of tokens and lowercase the tokens. remove the stopwords, using this list of stopwords. stemming. Use a stemmer in NLTK.When you parse the documents, you may only look at the subject and body. The subject line is indicated by the keyword Subject:, while the number of lines in the body is indicated by Lines: xx, which are the last xx lines of the document. Some files may miss the Lines: xx or have other exceptions. Please manually add the Lines: xx line for these few files and appropriately handle the exceptions. If you believe other fields of the documents might also be useful, you are free to include them.1/4