Shuchen Zhang’s Personal Blog

Graphic Neural Network Model for Screening of Novel Inhibitors for SARS CoV 3C-like Protease

2021-12-11T00:00:00+00:00

Introduction

This project aims to build a model that would aid novel drug design processes and is hosted here: https://github.com/susanzhang233/mollykill_2.0. Somewhat related to this mollykill 1.0, this project hopes to simplify limitations of the generator and decoder by disregarding GAN’s over-complicated generative structure. In this 2.0 version, we’ll be more focused on employing the accuracy and efficiency of the discriminator. Then, instead of letting the generator to come up with new molecules starting from zero. We’ll be applying a larger real world molecule datasets(ie. Zinc15), to mimic the traditional virtual/actual screening process to come up with potential inhibitors.

Model Structure

Featurizer

To represent the molecules in computer understandable format, this project uses the MolGraphConvFeaturizer from deepchem package that could be referred from here. This featurizer concatenates each molecule’s multiple features into two arrays: nodes array and edges array. Within each of the two arrays, there are then numbers of one-hot encoded arrays corresponding to numbers of atoms in each molecules, i.e. each atom and each bond is represented by one array.

To ensure that the size of the molecule representations are the same(i.e. to standardize input size), this project went one step further to sum up the atom and bond arrays along each column, therefore ending up with node arrays of length 30 and edges array of 11 for each molecule.

Dataset

The example dataset used for demonstration of this model is originally from PubChem AID1706, previously handled by JClinic AIcure team at MIT into this binarized label form. The dataset is also hosted in the data folder of this project.

Repository Explanation

FancyModule.py contains the major functions of this project
example.ipynb is an example usage in the model pipeline

Setup

If you are running locally(i.e. in a jupyter notebook in conda, just make sure you installed:

RDKit
DeepChem 2.5.0 & above
Tensorflow 2.4.0 & above

Then, please skip the following part and continue from Data Preparations.

To increase efficiency, we recommend running in Colab.

Then, we’ll first need to run the following lines of code, these will download conda with the deepchem environment in colab.

#!curl -Lo conda_installer.py https://raw.githubusercontent.com/deepchem/deepchem/master/scripts/colab_install.py
#import conda_installer
#conda_installer.install()
#!/root/miniconda/bin/conda info -e

#!pip install --pre deepchem
#import deepchem
#deepchem.__version__

Data Preparations

Now we are ready to import some useful functions/packages, along with our model.

Import Data

import FancyModule##our model

from rdkit import Chem
from rdkit.Chem import AllChem

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import deepchem as dc 

Select subset for training

Here, for demonstration, we’ll be selecting a subset of size $2405$ from this in-vitro assay that detects inhibition of SARS-CoV 3CL protease via fluorescence.

The dataset is originally from PubChem AID1706, previously handled by JClinic AIcure team at MIT into this binarized label form.

df = pd.read_csv('AID1706_binarized_sars.csv')

df_false = df[df['activity']==0].sample(n=2000, replace = False)#s
df_true = df[df['activity'] == 1]
df_subset = pd.concat([df_false, df_true], ignore_index = True)

df_subset

	smiles	activity
0	C1=CC=C(C(=C1)C(=O)O)N=NC2=C(NC3=C2C=C(C=C3)[N...	0
1	COC1=CC=CC=C1C2=NN(C(=O)C=C2)CC(=O)NC3CCC4=CC=...	0
2	CN(C)C(=S)NC1=CC=C(C=C1)CC2=CC=NC=C2	0
3	CC1=CC(=NC2=CC=CC=C12)SCC(=O)C3=C(N(C(=C3)C)C)C	0
4	CN(CC1=CC(=NN1)COC)C(=O)C2CCC(=O)N(C2)CC3=CC(=...	0
...	...	...
2400	C1COC2=C(O1)C=CC(=C2)NC(=O)C3=C(OC=N3)C4=CC=CC=C4	1
2401	COC(=O)C1=CC=CC=C1NC(=O)C2=CC3=C(C=C2)OCCCO3	1
2402	COC1=CC=CC=C1CCNC(=O)C(=O)NCC2N(CCO2)S(=O)(=O)...	1
2403	CN(C)CCNC(=O)C(=O)NCC1N(CCO1)S(=O)(=O)C2=CC=C(...	1
2404	C1COC(N1S(=O)(=O)C2=CC3=C(C=C2)OCCO3)CNC(=O)C(...	1

2405 rows × 2 columns

Observe the dataframe above, it contains a ‘smiles’ column, which stands for the smiles representation of the molecules. There is also an ‘activity’ column, in which it is the label specifying whether that molecule is considered as hit for the protein.

Set Minimum Length for molecules

Since we’ll be using graphic representation for each molecules, we’ll first need to cast the molecules into one universal length to fit into our training model. Our module contains a preparation function prepare_data that eliminates molecules shorter than the desired size.

df_minlength, y = FancyModule.prepare_data(df_subset, 12)

Featurization

Our module also have a featurize function that represents the molecules in graphic format that is supported by our model. The main structure of the featurizer could be referred from here.

Note: the original featurizer was modified for usage in this project.

Supported with input_df, the function would return featurized node and edges.

nodes, edges = FancyModule.featurize(df_minlength['smiles'])

Training

Now, we’re finally ready for training. We’ll first import some necessary functions from tensorflow.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

Here we’ll first initiate a model with the corresponding make_model functions in the package.

model = FancyModule.make_model()

Now we’ll train the model with the above subset and plot out the training curve.

model.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])

history = model.fit([np.array(nodes), np.array(edges)],
                    y,
                    epochs=80, 
                    verbose = False,
                    #class_weight=class_weights
                    #steps_per_epoch = 100,
                    )

plt.plot(history.history["accuracy"])
plt.gca().set(xlabel = "epoch", ylabel = "training accuracy")

[Text(0.5, 0, 'epoch'), Text(0, 0.5, 'training accuracy')]

Observe the training curve, the accuracy has a steady growing trend, and the model ultimately achieved an accuracy of around 86 percent.

Testing (Payoff)

Finally, we’ll demonstrate the testing phase(thoroughput screening) with a newly random selected subset from the above mentioned SARS-CoV 3CL protease assay.

df_false = df[df['activity']==0].sample(n=2000, replace = False)#s

df_test_subset = pd.concat([df_false, df_true], ignore_index = True)

The testing function accepts three parameters: the testing column of molecules in smiles format, trained model, and True/False statement of whether to save the result in the directory(default False).

FancyModule.test(df_test_subset['smiles'], model)

	smiles	prediction
1756	C([C@@H]1[C@H]([C@@H]([C@H]([C@H](O1)O[C@H]([C...	0.999190
2055	C1=CC=C2C(=C1)C(=CC(=C2N)C(C(F)(F)F)(C(F)(F)F)...	0.998951
2093	C1=CC(=CC(=C1)NC(=O)NCCNC(=O)C2=C(C=CC(=C2)OCC...	0.995320
2234	C1=CC(=CC=C1/C(=N/S(=O)(=O)C2=CC(=CC(=C2)C(F)(...	0.993566
1713	CC(C)CC(COCC1=CC=C(C=C1)C(F)(F)F)N2CCN(CCC2=O)...	0.991890
...	...	...
202	CC1CC2C3CCC4CC(CCC4(C3=CCC2(C1(C(=O)C)O)C)C)O	0.000008
125	CC#CC[N+](C)(CC#CC)CC(=O)C1=CC=CC=C1.[Br-]	0.000004
501	CCOC(=O)N1CCN(CC1)C(=O)C2CCC(CC2)C(=O)N3CCN(CC...	0.000003
540	CC(=O)OC1C2CCC1C(CC2)[N+]3(CCCCC3)C.[I-]	0.000002
454	CC1(CC(CC(N1)(C)C)N2CN(CC2=O)C3CCCCC3)C	0.000002

2405 rows × 2 columns

Yay, now we have our output data table. It has two columns, smiles format of molecules, and their corresponding prediction of inhibition score to the protein target, as estimated by our model.

Limitations(Future work)

Currently, the model accuracy is around 86-88%. More attempts with different model structure might be tested for possible improvements.
The efficiency of the featurization and training process might be improved.
Future wet-lab experiments could be done for confirmation from another aspect.

Acknowledgements

Featurizer https://deepchem.readthedocs.io/en/latest/api_reference/featurizers.html#deepchem.feat.MolGraphConvFeaturizer
Featurizer paper https://arxiv.org/pdf/1603.00856.pdf

Generative Model of Novel Inhibitor for SARS CoV 3C-like Protease

2021-12-09T00:00:00+00:00

Introduction

This project is aimed to build a Graphic GAN model that would aid drug design processes. The project is also hosted on github in the following link: https://github.com/susanzhang233/mollykill. For demonstration, the project is expected to learn graphical features of the molecules that are experimentally tested with inhibition effect for the specific protein SARS coronavirus 3C-like Protease (3CLPro) . Then, the model would develop a reasonable way to generate potential novel molecules inhibitors’ graphically representations. After that, with a defeaturizer, the graphical representations would be converted into visualizable molecule formats.

Intro to GAN

GAN, standing for Generative Adversarial Network, is commonly used in graphic works and more, such as generation of faces, music pieces. Here’s a great introduction. In this project, we’ll be building a GAN model that ultilizes the mathematical concepts of graphs in the chemical world.

Dataset

The dataset used for demonstration of this model is originally from PubChem AID1706, previously handled by JClinic AIcure team at MIT into this binarized label form. The dataset is also hosted here within this project.

Repository Explaination

model.py contains the source codes for the GAN model created
example.ipynb exemplifies the model usage

Demonstration:

Setup

If you are running this generator locally(i.e. in a jupyter notebook in conda, just make sure you installed:

RDKit
DeepChem 2.5.0 & above
Tensorflow 2.4.0 & above

Then, please skip the following part and continue from Data Preparations.

To increase efficiency, we recommend running this molecule generator in Colab.

Then, we’ll first need to run the following lines of code, these will download conda with the deepchem environment in colab.

#!curl -Lo conda_installer.py https://raw.githubusercontent.com/deepchem/deepchem/master/scripts/colab_install.py
#import conda_installer
#conda_installer.install()
#!/root/miniconda/bin/conda info -e

#!pip install --pre deepchem
#import deepchem
#deepchem.__version__

Data Preparations

Now we are ready to import some useful functions/packages, along with our model.

Import Data

import model##our model

from rdkit import Chem
from rdkit.Chem import AllChem

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import deepchem as dc 

Then, we are ready to import our dataset for training.

Here, for demonstration, we’ll be using this dataset of in-vitro assay that detects inhibition of SARS-CoV 3CL protease via fluorescence.

The dataset is originally from PubChem AID1706, previously handled by JClinic AIcure team at MIT into this binarized label form.

df = pd.read_csv('AID1706_binarized_sars.csv')

Observe the data above, it contains a ‘smiles’ column, which stands for the smiles representation of the molecules. There is also an ‘activity’ column, in which it is the label specifying whether that molecule is considered as hit for the protein.

Here, we only need those 405 molecules considered as hits, and we’ll be extracting features from them to generate new molecules that may as well be hits.

true = df[df['activity']==1]

Set Minimum Length for molecules

Since we’ll be using graphic neural network, it might be more helpful and efficient if our graph data are of the same size, thus, we’ll eliminate the molecules from the training set that are shorter(i.e. lacking enough atoms) than our desired minimum size.

num_atoms = 6 #here the minimum length of molecules is 6

input_df = true['smiles']
df_length = []
for _ in input_df:
    df_length.append(Chem.MolFromSmiles(_).GetNumAtoms() )

true['length'] = df_length #create a new column containing each molecule's length

true = true[true['length']>num_atoms] #Here we leave only the ones longer than 6
input_df = true['smiles']
input_df_smiles = input_df.apply(Chem.MolFromSmiles) #convert the smiles representations into rdkit molecules

Now, we are ready to apply the featurizer function to our molecules to convert them into graphs with nodes and edges for training. The logic behind the featurizer is to convert each molecule into an adjacency matrix storing bond informations and a node array with each atom’s features(here we only included atomic number).

#input_df = input_df.apply(Chem.MolFromSmiles) 
train_set = input_df_smiles.apply( lambda x: model.featurizer(x,max_length = num_atoms))

train_set

    ([6, 6, 6, 6, 6, 8], [[0, 1, 0, 0, 0, 0], [1, ...
    ([6, 6, 6, 6, 6, 6], [[0, 1, 0, 0, 0, 0], [1, ...
    ([6, 6, 6, 6, 6, 6], [[0, 1, 0, 0, 0, 0], [1, ...
    ([6, 6, 6, 6, 6, 6], [[0, 1, 0, 0, 0, 0], [1, ...
    ([6, 6, 7, 6, 7, 6], [[0, 1, 0, 0, 0, 0], [1, ...
                             ...                        
  ([6, 6, 8, 6, 6, 8], [[0, 1, 0, 0, 0, 1], [1, ...
  ([6, 8, 6, 8, 6, 6], [[0, 1, 0, 0, 0, 0], [1, ...
  ([6, 8, 6, 6, 6, 6], [[0, 1, 0, 0, 0, 0], [1, ...
  ([6, 7, 6, 6, 6, 7], [[0, 1, 0, 0, 0, 0], [1, ...
  ([6, 6, 8, 6, 7, 16], [[0, 1, 0, 0, 1, 0], [1,...
Name: smiles, Length: 405, dtype: object

We’ll take one more step to make the train_set into separate nodes and edges, which fits the format later to supply to the model for training

nodes_train, edges_train = list(zip(*train_set) )

Training

Now, we’re finally ready for generating new molecules. We’ll first import some necessay functions from tensorflow.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

Here we’ll first initiate a discriminator and a generator model with the corresponding functions in the package.

disc = model.make_discriminator(num_atoms)

The pipeline of the discriminator is shown below:

gene = model.make_generator(num_atoms, noise_input_shape = 100)

The pipeline of the generator is shown below:

Then, with the train_batch function, we’ll supply the necessary inputs and train our network. Upon some experimentations, an epoch of around 160 would be nice for this dataset.

generator_trained = model.train_batch(
                                disc, gene, 
                                np.array(nodes_train), np.array(edges_train), 
                                noise_input_shape = 100, EPOCH = 160, BATCHSIZE = 2, 
                                plot_hist = True, temp_result = False
    )

There are two possible kind of failures regarding a GAN model: model collapse and failure of convergence. Model collapse would often mean that the generative part of the model wouldn’t be able to generate diverse outcomes. Failure of convergence between the generative and the discriminative model could likely way be identified as that the loss for the discriminator has gone to zero or close to zero.

Observe the above generated plot, in the upper plot, the loss of discriminator has not gone to zero/close to zero, indicating that the model has possibily find a balance between the generator and the discriminator. In the lower plot, the accuracy is fluctuating between 1 and 0, indicating possible variability within the data generated.

Therefore, it is reasonable to conclude that within the possible range of epoch and other parameters, the model has successfully avoided the two common types of failures associated with GAN.

Rewarding Phase

The above train_batch function is set to return a trained generator. Thus, we could use that function directly and observe the possible molecules we could get from that function.

no, ed = generator_trained(np.random.randint(0,20
                                             
                                            , size =(1,100)))#generated nodes and edges

abs(no.numpy()).astype(int).reshape(num_atoms), abs(ed.numpy()).astype(int).reshape(num_atoms,num_atoms)

(array([6, 9, 5, 7, 8, 6]),
 array([[0, 0, 0, 0, 0, 0],
        [1, 0, 2, 0, 0, 0],
        [0, 2, 0, 2, 0, 0],
        [0, 0, 2, 0, 2, 0],
        [0, 0, 0, 2, 0, 3],
        [2, 0, 0, 0, 3, 0]]))

With the de_featurizer, we could convert the generated matrix into a smiles molecule and plot it out=)

cat, dog = model.de_featurizer(abs(no.numpy()).astype(int).reshape(num_atoms), abs(ed.numpy()).astype(int).reshape(num_atoms,num_atoms))

Chem.MolToSmiles(cat)

'C#O=N=B=FC'

Here are some examples of molecules generated by the model. Though some molecules may appear to be not legit, a similar trend among these molecules preliminarily demonstrates that we have get the model working.

Brief Result Analysis

from rdkit import DataStructs

With the rdkit function of comparing similarities, here we’ll demonstrate a preliminary analysis of the molecule we’ve generated. With “CCO” molecule as a control, we could observe that the new molecule we’ve generated is more similar to a random selected molecule(the fourth molecule) from the initial training set.

This may indicate that our model has indeed extracted some features from our original dataset and generated a new molecule that is relevant.

DataStructs.FingerprintSimilarity(Chem.RDKFingerprint(Chem.MolFromSmiles("[Li]NBBC=N")), Chem.RDKFingerprint(Chem.MolFromSmiles("CCO")))# compare with the control

0.0

#compare with one from the original data
DataStructs.FingerprintSimilarity(Chem.RDKFingerprint(Chem.MolFromSmiles("[Li]NBBC=N")), Chem.RDKFingerprint(Chem.MolFromSmiles("CCN1C2=NC(=O)N(C(=O)C2=NC(=N1)C3=CC=CC=C3)C")))

0.017079419299743808

Limitations(Future work)

Currently, the generation of molecules is limited to a specific length that is shorter than most molecules in the real world. Future work involving some concepts of Conditional GAN might be employed.
The dimension of the discriminator might be improved by adding more features of the molecules(ie. hybridization, stereochemistry, etc)
The efficiency of the featurizer in treating edges informations might be improved(viable representation of rings, etc)

Acknowledgements

Where I’ve obtained my dataset: https://github.com/yangkevin2/coronavirus_data/tree/master/data
A nice introduction video of GAN structure: https://github.com/whoIsTheGingerBreadMan/YoutubeVideos
A nice introduction blogpost: https://machinelearningmastery.com/how-to-develop-a-conditional-generative-adversarial-network-from-scratch/
MolGAN, a great previous work of related field: https://arxiv.org/pdf/1805.11973.pdf

Project Reflection

2021-06-11T00:00:00+00:00

Overview

My project, Mollykill, aims to comprehensively apply the informations regarding PIC16B and beyond in crafting a basic generative molecular model and providing a basic idea for computational methodologies in the process of drug design.

Project Achievements

Currently, my project provides a basic pipeline of the process of generating new molecules that possibly possess similar features with the original dataset.

The two specific aspects I like about my project is first how it managed to get the structure of a GAN that seems to be successful, and is able to generate molecules that at least looks viable. Secondly, the featurizer for the model is able to give out a some what reasonable graph representation of the molecules. Generally, I enjoy observing that computational approaches could be intervened with the world of other subjects to generate great outcomes.

Future work

Currently, the generation of molecules is still limited to a specific length that is shorter than most molecules in the real world. If users want to generate longer molecules, the generated molecule would not be syntactically recognizable by the defeaturizer. Future work involving some concepts of Conditional GAN that ensures the viability of the generated molecules might be employed.
The dimension of the discriminator might be improved by adding more features of the molecules(ie. hybridization, stereochemistry, etc)

Proposed & Outcome

Indeed, the rationale of the project has changed a bit through time compared to that in the original proposal. Initially, I’ve thought of making a classification model and applying a larger database. In other words, creating a pipeline of computational virtual screening process. However, generating actual novel molecules with preliminary wet-lab assy informations is indeed very important in drug discovery, the molecules generated by computer might even have better effect than those from the initial molecule database. Thus, I ultimately decided to build a generative model for molecules. Other technical details, such as the final present format, has not changed too much.

Lessons Learned!

Thanks to the help from Professor and Erin, I’ve learned a lot about github, including how to fork other’s repository, git clone, etc. When building my deep learning model, I’ve also gained more understanding about the math concepts when trying to understand what those layers of tensorflow do. Furthermore, I’ve also gotten more used to referring to the API of all sort of functions when making use of them.

Beyond and Above

Regarding the specific skills, such as tensorflow and git, learned during doing this project, the experience would definitely help me to adapt more efficiently when encountering related topics in the future. Moreover, an important skill that Sthis course is taught me is to self-learn. I’ve gradually got use to browsing youtube videos, tutorial blogposts, github source codes, and function APIs for stuffs I need. Most importantly, through crafting this project, I’ve somehow determined my future career interest, in which I would like to examine to what extend could computational methods aid the development in the field of traditional health science discoveries.

Overall, special thanks to Professor Chodrow and TA Erin for supporting us emotionally and technically during this quarter about this project, this course, and more beyond. Even though I don’t have a groupmate, I would also like to thank all my classmates, for that I would always be motivated when thinking that we are progressing as a whole team.

Blog Post 3 - Fake News Classifier

2021-05-18T00:00:00+00:00

In this blog, we’ll build a machine learning model with Tensorflow that helps us to classify fake news.

First, we’ll import some of the needed packages our dataset to feed the model.

import pandas as pd
import tensorflow as tf
import numpy as np

from matplotlib import pyplot as plt

Both the training and testing datasets are accessed from Kaggle:

#train_url = "https://github.com/PhilChodrow/PIC16b/blob/master/datasets/fake_news_train.csv?raw=true"
df = pd.read_csv('fake_news_train.csv')

test_url = "https://github.com/PhilChodrow/PIC16b/blob/master/datasets/fake_news_test.csv?raw=true"
test_df = pd.read_csv(test_url)

df

	Unnamed: 0	title	text	fake
0	17366	Merkel: Strong result for Austria's FPO 'big c...	German Chancellor Angela Merkel said on Monday...	0
1	5634	Trump says Pence will lead voter fraud panel	WEST PALM BEACH, Fla.President Donald Trump sa...	0
2	17487	JUST IN: SUSPECTED LEAKER and “Close Confidant...	On December 5, 2017, Circa s Sara Carter warne...	1
3	12217	Thyssenkrupp has offered help to Argentina ove...	Germany s Thyssenkrupp, has offered assistance...	0
4	5535	Trump say appeals court decision on travel ban...	President Donald Trump on Thursday called the ...	0
...	...	...	...	...
22444	10709	ALARMING: NSA Refuses to Release Clinton-Lynch...	If Clinton and Lynch just talked about grandki...	1
22445	8731	Can Pence's vow not to sling mud survive a Tru...	() - In 1990, during a close and bitter congre...	0
22446	4733	Watch Trump Campaign Try To Spin Their Way Ou...	A new ad by the Hillary Clinton SuperPac Prior...	1
22447	3993	Trump celebrates first 100 days as president, ...	HARRISBURG, Pa.U.S. President Donald Trump hit...	0
22448	12896	TRUMP SUPPORTERS REACT TO DEBATE: “Clinton New...	MELBOURNE, FL is a town with a population of 7...	1

22449 rows × 4 columns

Apart from the indices column, the dataframe contains three more informative columns: title column, the titles of the news; text column, the content of that news; the fake column, label indicating whether the news is 0(real), or 1(fake).

Make Dataset

Before we actually start the training process, transforming the dataset into decent forms for training are also very important. Here, we’ll create a make_dataset function that’ll do two things:

remove stop words from the article text and title columns(stop words refers to those most common words in a language, which could be filtered out for training. Some common stop words are a, the, as, at, by, to, etc)
construct a tensorflow dataset with inputs (title, text) and output fake column labels. Dataset format of enables the input to be smoothly handled by tensorflow (tensorflow dataset is a specific class containing many useful functions to keep you organized)

To remove stop words, we’ll first import some extra packages and functions.

the stop words function gives you a list of common stop words of specified language;
the word_tokenize function breaks a string into single words and punctuations of a 1d array form;
the TreebankWordDetokenizer function detokenizes a 1d array of words, combining it back into one individual string.

import nltk
#nltk.download('stopwords') #only need to be downloaded once
#nltk.download('punkt')  #only need to be downloaded once as well
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize 
stop = stopwords.words('english')

from nltk.tokenize.treebank import TreebankWordDetokenizer

def make_dataset(df):
    df_ = df
    
    #remove stopwords for the dataframe
    df_['title'] = df_['title'].apply(word_tokenize)# split a string into separate tokens
    df_['title'] = df['title'].apply(lambda x: [w for w in x if not w in stop] )#remove stopwords
    df_['text'] = df_['text'].apply(word_tokenize)
    df_['text'] = df_['text'].apply(lambda x: [w for w in x if not w in stop] )
    df_['text'] = df_['text'].apply(lambda x: TreebankWordDetokenizer().detokenize(x))#combine the tokens back to a complete sentence
    df_['title'] = df_['title'].apply(lambda x: TreebankWordDetokenizer().detokenize(x))

    #create tensorflow dataset with inputs (title, text) and output fake column labels and return it
    return tf.data.Dataset.from_tensor_slices(({"title": df_['title'], "text": df_['text']}, df_['fake']))

We then apply the function to our training and testing datasets:

dataset = make_dataset(df)

test_dataset = make_dataset(test_df)

Train Validation Split, Create Batches

Now we are ready to construct some train & validation split inside our training set. Along with that, we’ll also set batches for training set, as training with the original dataset volume might be too inefficient for demonstration purposes.

#shuffle the complete training set
dataset = dataset.shuffle(buffer_size = len(dataset))

#specify training and validation size: 0.7 & 0.2, respectively
train_size = int(0.7*len(dataset))
val_size   = int(0.2*len(dataset))

#pick out first 0.7 as training set, with a batch of 100
train = dataset.take(train_size).batch(100)
#pick out the next 0.2 as validation set, with a batch of 100
val   = dataset.skip(train_size).take(val_size).batch(100)
#test  = dataset.skip(train_size + val_size).batch(100) #last 0.1 of data

#check for the length of the two sets
len(train), len(val)#, len(test)

(158, 45)

test_ds = test_dataset.batch(100)

Create Models

We are now ready to make our models. While our data contains informative columns of both news title and news text, we would like to know whether titles only, text only, or both, would be more effective for detection of fake news. Therefore, we’ll be creating three models, with similar strategies, and comparing their accuracies.

Vectorization layer

No matter which combination we’ll be using, we all need to first vectorize the input strings. Vectorization refers to processing of the strings into computer readable format, which is, numbers. There are multiple ways of vectorization of strings, such as the term-document matrix, etc. Here, we’ll be using the TextVectorization function provided by tensorflow. This function would represent each of the words by its rank of frequency in the whole dataset. First, we’ll need to define a standardization function to supply to the TextVectorization function.

Here, the standardization function would convert all words to lowercases and remove all punctuations.

def standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    no_punctuation = tf.strings.regex_replace(lowercase,
                                  '[%s]' % re.escape(string.punctuation),'')
    return no_punctuation 

Then, with the TextVectorization function, we’ll make a vectorize_layer that transforms each words in the training set, after standardization treatment, into tensors with the words’ corresponding frequency rank.

from tensorflow.keras import losses
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

import re
import string

# only the first 2000 distinct words in the whole set will be tracked
max_tokens = 2000

# only the first 20 words of each headline will be considered
sequence_length = 20

vectorize_layer = TextVectorization(
    standardize=standardization,
    max_tokens=max_tokens, # only consider this many words
    output_mode='int',
    output_sequence_length=sequence_length) 

After creating the layers, we would still need to adapt the layer to our set. By adapting it, we will ensure that the vectorization layer have the words’ corresponding frequency ranks fixed. Thus whenever we are training or testing with that layer, each distinct word would have that one and only ranking.

vectorize_layer.adapt(train.map(lambda x, y: x["title"]))
vectorize_layer.adapt(train.map(lambda x, y: x["text"]))

Then, we’ll specify two input layers for title and text.

from tensorflow import keras
from tensorflow.keras import layers

title_input = keras.Input(
    shape = (1,), 
    name = "title",
    dtype = "string"
)

text_input = keras.Input(
    shape = (1,), 
    name = "text",
    dtype = "string"
)

Model trains with titles only

Since we have two kinds of inputs, title and text, we’ll be using the keras functional API, instead of sequential API. The functional API can create layers that have more flexible sequences, therefore suitable for building a model with multiple inputs. First, we’ll build a model that learns the news titles only.

Our first layer would be the vectorize_layer we just built. In each of the following layers, we supply the input as the result obtained from the preceding layer in (). With Dropout, the input units in that layer will be set to 0 with a frequency of rate at each step during training time, which helps prevent overfitting. The Embedding layer creates a vocabulary space, giving each word its own unique spot in that space. Furthermore, the spots of the words are set in a way that the distance and direction between words would indicate relatedness and relationships. We’ll give this layer a name “embedding_tl” to ease the process of examining it later.

x = vectorize_layer(title_input)
x = layers.Embedding(max_tokens, output_dim = 5, name = "embedding_tl")(x)
x = layers.Dropout(0.2)(x)
x = layers.GlobalAveragePooling1D()(x)
x = layers.Dropout(0.2)(x)
x = layers.Dense(32, activation='relu')(x)
output = layers.Dense(2, name = "fake")(x)

Now, we are finally ready to build the model. Only specifying the initial input, tensorflow will be able to extract all that information we’ve provided of the layers.

model_tl = keras.Model(
    inputs = title_input,
    outputs = output
)

model_tl.summary()

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
title (InputLayer)           [(None, 1)]               0         
_________________________________________________________________
text_vectorization (TextVect (None, 20)                0         
_________________________________________________________________
embedding_tl (Embedding)     (None, 20, 5)             10000     
_________________________________________________________________
dropout (Dropout)            (None, 20, 5)             0         
_________________________________________________________________
global_average_pooling1d (Gl (None, 5)                 0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 5)                 0         
_________________________________________________________________
dense (Dense)                (None, 32)                192       
_________________________________________________________________
fake (Dense)                 (None, 2)                 66        
=================================================================
Total params: 10,258
Trainable params: 10,258
Non-trainable params: 0
_________________________________________________________________

As always, we’ll compile the model and fit it to the training set.

model_tl.compile(optimizer = "adam",
              loss = losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy']
)

history = model_tl.fit(train, 
                    validation_data=val,
                    epochs = 50, 
                    verbose = False)

And we’ll plot out the accuracy curve.

plt.plot(history.history["accuracy"], label = "training")
plt.plot(history.history["val_accuracy"], label = "validation")
plt.gca().set(xlabel = "epoch", ylabel = "accuracy")
plt.legend()

model_tl.evaluate(test_ds)

225/225 [==============================] - 0s 1ms/step - loss: 0.0840 - accuracy: 0.9752

[0.08402662724256516, 0.9751881957054138]

Evaluating the model with only titles with out test set, we could see that the accuracy is actually pretty high: 0.97! Let’s see if the other two models could do better.S

Model trains with text only

Training with text would have a similar process: specify each layers, make the model, compile it, fit it, evaluate and observe.

x_ = vectorize_layer(text_input)
x_ = layers.Embedding(max_tokens, output_dim = 5, name = "embedding_tx")(x_)
x_ = layers.Dropout(0.2)(x_)
x_ = layers.GlobalAveragePooling1D()(x_)
x_ = layers.Dropout(0.2)(x_)
x_ = layers.Dense(32, activation='relu')(x_)
output = layers.Dense(2, name = "fake")(x_)

model_tx= keras.Model(
    inputs = text_input,
    outputs = output
)

model_tx.compile(optimizer = "adam",
              loss = losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy']
)

model_tx.fit(train, 
                    validation_data=val,
                    epochs = 50, 
                    verbose = False)

model_tx.evaluate(test_ds)

225/225 [==============================] - 1s 6ms/step - loss: 0.1612 - accuracy: 0.9463

[0.16124901175498962, 0.9462782144546509]

The accuracy for the model trained with the text only has a lower accuracy. Somehow reasonable, as the text is longer, and a fake news might also possess some sentences that sounds true.

Model trained with both text and title

both = layers.concatenate([x, x_], axis = 1)
output_both = layers.Dense(2, name = "fake")(both)

model_both= keras.Model(
    inputs = [title_input, text_input],
    outputs = output_both
)

model_both.summary()

Model: "model_2"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
title (InputLayer)              [(None, 1)]          0                                            
__________________________________________________________________________________________________
text (InputLayer)               [(None, 1)]          0                                            
__________________________________________________________________________________________________
text_vectorization (TextVectori (None, 20)           0           title[0][0]                      
                                                                 text[0][0]                       
__________________________________________________________________________________________________
embedding_tl (Embedding)        (None, 20, 5)        10000       text_vectorization[0][0]         
__________________________________________________________________________________________________
embedding_tx (Embedding)        (None, 20, 5)        10000       text_vectorization[1][0]         
__________________________________________________________________________________________________
dropout (Dropout)               (None, 20, 5)        0           embedding_tl[0][0]               
__________________________________________________________________________________________________
dropout_2 (Dropout)             (None, 20, 5)        0           embedding_tx[0][0]               
__________________________________________________________________________________________________
global_average_pooling1d (Globa (None, 5)            0           dropout[0][0]                    
__________________________________________________________________________________________________
global_average_pooling1d_1 (Glo (None, 5)            0           dropout_2[0][0]                  
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 5)            0           global_average_pooling1d[0][0]   
__________________________________________________________________________________________________
dropout_3 (Dropout)             (None, 5)            0           global_average_pooling1d_1[0][0] 
__________________________________________________________________________________________________
dense (Dense)                   (None, 32)           192         dropout_1[0][0]                  
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 32)           192         dropout_3[0][0]                  
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 64)           0           dense[0][0]                      
                                                                 dense_1[0][0]                    
__________________________________________________________________________________________________
fake (Dense)                    (None, 2)            130         concatenate[0][0]                
==================================================================================================
Total params: 20,514
Trainable params: 20,514
Non-trainable params: 0
__________________________________________________________________________________________________

model_both.compile(optimizer = "adam",
              loss = losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy']
)

model_both.fit(train, 
                    validation_data=val,
                    epochs = 50, 
                    verbose = False)

<tensorflow.python.keras.callbacks.History at 0x7f84c388e910>

model_both.evaluate(test_ds)

225/225 [==============================] - 1s 6ms/step - loss: 0.0371 - accuracy: 0.9913

[0.037129346281290054, 0.991313636302948]

Here, we could observe that training with both text and title yields an accuracy of 0.9913!

Embedding Analysis

In the next stage, we’ll extract the previous Embedding layer, do some visualization of it to demonstrate how the words are learned by the model. We’ll plot out each word’s spot in the whole 2d vocabulary space. Since our embedding layer has an output of 5 dimensions, we’ll first need to conduct principle component analysis to transform it into 2d representations. Sci-kit learn has that function created for us.

First, we’ll get the vocabularies and weights corresponding to the words in the two embedding layers of title/text out:

weights_tl = model_both.get_layer('embedding_tl').get_weights()[0]
weights_tx = model_both.get_layer('embedding_tx').get_weights()[0]# get the weights from the embedding layer
vocab = vectorize_layer.get_vocabulary()  

vocab = vectorize_layer.get_vocabulary()  

weights_tl

array([[ 0.10268341,  0.09916885, -0.11284392,  0.12111397, -0.10344727],
       [-0.10890676, -0.13384679,  0.10487777, -0.12676527,  0.13442   ],
       [ 0.05327572,  0.05760193, -0.11840718, -0.00806848, -0.11700375],
       ...,
       [ 0.02861674,  0.02574723,  0.04686182,  0.09227268, -0.03490727],
       [ 0.15639482,  0.09743544, -0.06591856,  0.02370312, -0.01798774],
       [ 0.00066107, -0.01369754, -0.06120993,  0.03191474, -0.09007049]],
      dtype=float32)

Next, we perform PCA to the weights to convert them into 2d arrays:

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
weights_tl = pca.fit_transform(weights_tl)
weights_tx = pca.fit_transform(weights_tx)

len(vocab)

Then, we’ll create a data frame containing the embedding of words in titles.

embedding_tl = pd.DataFrame({
    'word' : vocab, 
    'x0'   : weights_tl[:,0],
    'x1'   : weights_tl[:,1]
})
embedding_tl

	word	x0	x1
0		-0.195497	0.003443
1	[UNK]	0.317627	0.025609
2	said	-0.105505	-0.010339
3	trump	0.100054	-0.045148
4	the	1.743519	-0.003649
...	...	...	...
1995	tonight	0.910266	0.059968
1996	repeated	0.170522	0.024102
1997	projects	-0.014011	-0.052086
1998	outcome	-0.117215	0.046006
1999	launch	-0.029681	-0.028275

2000 rows × 3 columns

We’ll do the same thing with the embedding of words in the news text.

embedding_tx = pd.DataFrame({
    'word' : vocab, 
    'x0'   : weights_tx[:,0],
    'x1'   : weights_tx[:,1]
})

Now, we are finally ready to plot the words and their embeddings out！We’ll be using plotly express function!

import plotly.express as px 
import plotly.io as pio
pio.templates.default = "plotly_white"
import plotly.express as px 

fig = px.scatter( embedding_tl, 
                 x = "x0", 
                 y = "x1", 
                 size = list(np.ones(len(embedding_tl))),
                 size_max = 2,
                 hover_name = "word")

fig.show()

Here, the graph demonstrates each words’ spot in the vocabulary space.

from plotly.io import write_html

write_html(fig, "emb_tx.html")