Introduction
Have you ever wondered how to tell if a text was written by a human or an AI? Do you think you could tell the difference? ‘Can you tell me about the history of the Kohinoor (Koh-i-Noor) Diamond?’. Try to guess who generated this question: A human or an AI. With the advances in natural language generation, it is becoming harder and harder to tell the difference. In this article, we will build a machine-learning model that will tell if a human or ChatGPT generated the text. Sounds interesting, right? Let’s get started.
This article was published as a part of the Data Science Blogathon.
Table of Contents
ChatGPT
ChatGPT is a conversational AI or machine learning system. It listens, learns, and challenges. This system takes a natural language text as input and provides responses. It’s just like you are talking with a person with more intelligence than you who can answer almost all your queries. Do you even imagine that we will get something like this in the future? This ChatGPT was developed by OpenAI and launched in November 2022. It uses an iterative process that considers the context of user input, takes from previously viewed conversations, and makes use of its collective understanding of those conversations to provide a new response. It uses human feedback to minimize harmful, untruthful, or biased answers.
Now let’s see some examples to understand how this ChatGPT responds.
User: What do you think of pineapple on pizza?
ChatGPT: I think pineapple on pizza is an interesting combination of flavors. Some people may like it and some may not. It depends on personal taste.
User: I like cats more than dogs.
ChatGPT: I see. Why do you like cats more than dogs?
User: I think climate change is a serious problem.
ChatGPT: I agree with you. Climate change is a serious problem that affects everyone on this planet.
Let’s start building our model.
Problem Statement: We will build a machine-learning model that can classify text as human-written or ChatGPT generated. It should be able to handle multiple types of texts like questions, essays, stories, jokes, code, etc.
Dataset Description
- Here is the link to the dataset that I have used in this project: Dataset
- This dataset contains four columns: text, paraphrases, category, and source.
- These texts were taken from different sources.
- Each of these texts was paraphrased to produce five more texts.
- It has 419197 rows and 4 columns.
Machine Learning Models Implementation
Let’s start by importing basic necessary machine learning libraries such as numpy and pandas.
import numpy as np
import pandas as pdUse pandas to load the dataset and generate a data frame.
df=pd.read_csv(r'C:\Users\Admin\Downloads\chatgpt_paraphrases.csv')
print(df)
print(df.shape)
#(419197, 4)
Let’s view one text and corresponding its paraphrase to understand the dataset. In the output, we can see the text is a question asking for the story of the Kohinoor diamond. The same sentence was paraphrased in five different ways by ChatGPT.
df['text'][1]
df['paraphrases'][1]
Create a dictionary with texts and their corresponding categories like determining whether the text is generated by humans or ChatGPT. It will be classified based on whether it’s a text or a paraphrase. If it is a text then it’s human-generated else if it is a paraphrase then it’s ChatGPT generated. If you view this category, you will get a dictionary as shown in the image.
category={}
for i in range(len(df)):
chatgpt=df.iloc[i]["paraphrases"][1:-1].split(', ')
for j in chatgpt[:1]:
category[j[1:-1]]='chatgpt'
category[df.iloc[i]['text']]="human"
category
Convert this category dictionary into a data frame using pandas. Create two columns as text and category where the category has two unique values such as human and ChatGPT. Shuffle all the rows in the data frame to avoid overfitting. We will take the first 20000 rows after shuffling to make it easy. Let’s view the data frame created.
df=pd.DataFrame(category.items(),columns=["text","category"])
df=df.sample(frac=1)
df=df[:20000]
df
View the unique values of humans and ChatGPT. There are 10340 texts generated by humans and 9660 texts generated by ChatGPT.
df["category"].value_counts()
Take two array variables X and Y. X will have a text column of the data frame and Y will have a category column of the data frame. This basically is the input and output for the model. We will give X as input and it will predict Y as output.
X=df['text']
y=df['category']Split the dataset into train and test datasets
Next split the entire dataset using train_test_split into X_train, X_test, y_train, and y_test for further processing.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)Let’s vectorize them using a TF-IDF vectorizer. It is a numerical statistic that reflects how important a word is to a document in a collection or corpus. In information retrieval and text mining, it is frequently employed as a weighting factor. For this import TfidfVectorizer.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)Selecting Best Classifier
Instead of taking one specific classifier and building it, let’s take some set of classifiers and calculate their accuracy score and f1 score to know the best classifier among them. Here we used logistic regression, support vector classifier, decision tree classifier, voting classier, KNN classifier, Random forest, Extra trees, Adaboost, bagging classifier, and gradient boosting classifier.
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import GradientBoostingClassifier
lg = LogisticRegression(penalty='l1',solver='liblinear')
sv = SVC(kernel='sigmoid',gamma=1.0)
mnb = MultinomialNB()
dtc = DecisionTreeClassifier(max_depth=5)
knn = KNeighborsClassifier()
rfc = RandomForestClassifier(n_estimators=50,random_state=2)
etc = ExtraTreesClassifier(n_estimators=50,random_state=2)
abc = AdaBoostClassifier(n_estimators=50,random_state=2)
bg = BaggingClassifier(n_estimators=50,random_state=2)
gbc = GradientBoostingClassifier(n_estimators=50,random_state=2)Now calculate the accuracy score and f1 score of these classifiers.
def prediction(model,X_train,X_test,y_train,y_test):
model.fit(X_train,y_train)
pr = model.predict(X_test)
acc_score = metrics.accuracy_score(y_test,pr)
f1= metrics.f1_score(y_test,pr,average="binary", pos_label="chatgpt")
return acc_score,f1
acc_score = {}
f1_score={}
clfs= {
'LR':lg,
'SVM':sv,
'DTC':dtc,
'KNN':knn,
'RFC':rfc,
'ETC':etc,
'ABC':abc,
'BG':bg,
'GBC':gbc,
}
for name,clf in clfs.items():
acc_score[name],f1_score[name]= prediction(clf,X_train_tfidf,X_test_tfidf,y_train,y_test)
#View those scores
acc_score
f1_score
If we compare these scores we got the highest for ExtraTrees Classifier (ETC). So we will use this extra trees classifier to train the model and predict the future. For that train the model using the fit method.
Machine Learning Model – Extra Trees Classifier
Extra Trees Classifier: An extra trees classifier is a type of ensemble learning method that uses multiple randomized decision trees to improve the predictive accuracy and control over-fitting. It is also called an Extremely Randomized Trees Classifier. It differs from classic decision trees in the way they are built. Normal decision trees have some drawbacks like overfitting and high variance. So to avoid them different ensemble learning methods are introduced. The extra trees classifier is one among them. Instead of using bootstrap samples, Extra Trees Classifier uses the whole original dataset for each tree, but with a random sampling of features for each split. And also instead of finding the optimal split point for each feature, Extra Trees Classifier randomly selects a split point from a uniform distribution within the feature’s range. This is a powerful machine learning technique that can handle complex classification problems with high accuracy and low overfitting.
etc.fit(X_train_tfidf,y_train)Confusion Matrix
Predict the test dataset and get a confusion matrix. This confusion matrix is basically a table that defines the performance of the algorithm. Here for our classification problem, it will give four values. False Positive (FP), True Positive (TP), False Negative (FN), and True Negative (TN) are the four values that it will give. We will also plot this confusion matrix. For that import matplotlib and seaborn for visualizations.
from sklearn.metrics import confusion_matrix
y_pred =etc.predict(X_test_tfidf)
cm = confusion_matrix(y_test, y_pred)
print(cm)
import seaborn as sn
import matplotlib.pyplot as plt
confusion_matrix = pd.DataFrame(cm, index = [i for i in ["ChatGPT","Human"]],
columns = [i for i in ["ChatGPT","Human"]])
plt.figure(figsize = (20,14))
sn.heatmap(confution_matrix, annot=True,cmap="YlGnBu", fmt='g')
Now let’s see the predicted results by the model. In the previous step, We have predicted and saved the results in y_pred. This y_pred is an array. So convert it into a data frame using pandas and rename the column from 0 to ‘category predicted’ and view it.
y_preddf=pd.DataFrame(y_pred)
y_preddf.rename(columns={0:'category predicted'},inplace=True)
y_preddf
Comparison
To compare actual results and predicted results, let’s join the data frames. And view the data frame. Here I viewed rows from 20 to 30. In the results, you can see almost all are predicted correctly whereas some are incorrectly predicted. For the 23rd text, our model predicted that it was generated by a human but in actuality, it was generated by ChatGPT.
x_testdf=pd.DataFrame(X_test)
y_testdf=pd.DataFrame(y_test)
x_testdf['id'] = range(1, len(x_testdf) + 1)
y_testdf['id'] = range(1, len(y_testdf) + 1)
y_preddf['id'] = range(1, len(y_preddf) + 1)
join1=y_testdf.merge(x_testdf, how = 'inner' ,indicator=False)
join_df=join1.merge(y_preddf, how = 'inner' ,indicator=False)
join_df[20:30]
Finding Accuracy
So these incorrectly classified texts are false positives and false negatives. Let’s see the accuracy of our model. We got an accuracy of 78.7%. This can be improved by using a large number of rows while training and by increasing the number of epochs. We had taken only 20000 rows and 10 epochs for our experiment.
accuracy_score=metrics.accuracy_score(y_pred,y_test)*100
accuracy_score
#78.7Now it’s time to test our model with our own texts instead of texts from the dataset. I have given four different texts. First, first two texts gave humans even though I was given ChatGPT in the second example. This tells us how accurately our model is predicting. In the third example, I used some words like a step-by-step guide, and in the last example I provided a text like recommending some websites that resemble ChatGPT and our model also predicted it as ChatGPT.
input=['Hello!! This is Amrutha']
vect_input=vectorizer.transform(input)
etc.predict(vect_input)
#array(['human'], dtype=object)
input=['Hello!! This is chatgpt']
vect_input=vectorizer.transform(input)
etc.predict(vect_input)
#array(['human'], dtype=object)
input=['Can you please provide a step by step guide for writing articles on analytics vidhya']
vect_input=vectorizer.transform(input)
etc.predict(vect_input)
#array(['chatgpt'], dtype=object)
input=['These are the websites for watching movies that I can recommend you']
vect_input=vectorizer.transform(input)
etc.predict(vect_input)
#array(['chatgpt'], dtype=object)Conclusion
Nowadays the word that we are listening to more often is ChatGPT. This is literally being used everywhere including by students, engineers, writers, teachers, and all. Because of the potential it has in generating wonderful answers to their queries. These ChatGPT-generated texts exactly sound like humans and we humans cannot tell the difference just by looking at those texts. So to distinguish the texts generated by ChatGPT we have built a model which successfully tells either ‘human’ or ‘ChatGPT’.
- ChatGPT is an AI language model that can generate human-like text on various topics and tasks.
- We have used a dataset with a good number of both human and ChatGPT-generated texts to train our model.
- We have selected the best classifier which is the extra trees classifier to train our model.
- We need to be careful of the ethical and social implications of ChatGPT and the possibility of false positives and false negatives in our machine learning model.
Hope you found this article useful. Connect with me on LinkedIn.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
This is Amrutha, I am pursuing B.Tech in the Computer science Department. I am interested in developing ML Models with python and Data Analysis. And also I have an interest in Web Development. I hope my articles in Analytics Vidhya help you to learn better. Thank you!!
