Bring DevOps To Data Science With MLOps

# Report training set score
train_score = model.score(X_train, y_train) * 100

# Report test set score
test_score = model.score(X_test, y_test) * 100

# Write scores to a file
with open("metrics.txt", 'w') as outfile:
outfile.write("Training variance explained: %2.1f%%n" % train_score)
outfile.write("Test variance explained: %2.1f%%n" % test_score)

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, plot_roc_curve
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
df_heart = pd.read_csv('SAHeart.csv', index_col=0)
df_heart.head(10)
df_heart.describe()
df_heart.drop('famhist', axis=1, inplace=True)
# Set random seed
seed = 52
# Split into train and test sections
y = df_heart.pop('chd')
X_train, X_test, y_train, y_test = train_test_split(df_heart, y, test_size=0.2, random_state=seed)
# Build logistic regression model
model = LogisticRegression(solver='liblinear', random_state=0).fit(X_train, y_train)
# Report training set score
train_score = model.score(X_train, y_train) * 100
# Report test set score
test_score = model.score(X_test, y_test) * 100
# Write scores to a file
with open("metrics.txt", 'w') as outfile:
        outfile.write("Training variance explained: %2.1f%%n" % train_score)
        outfile.write("Test variance explained: %2.1f%%n" % test_score)
# Confusion Matrix and plot
cm = confusion_matrix(y_test, model.predict(X_test))
fig, ax = plt.subplots(figsize=(8, 8))
ax.imshow(cm)
ax.grid(False)
ax.xaxis.set(ticks=(0, 1), ticklabels=('Predicted 0s', 'Predicted 1s'))
ax.yaxis.set(ticks=(0, 1), ticklabels=('Actual 0s', 'Actual 1s'))
ax.set_ylim(1.5, -0.5)
for i in range(2):
    for j in range(2):
        ax.text(j, i, cm[i, j], ha='center', va='center', color='red')
plt.tight_layout()
plt.savefig("cm.png",dpi=120) 
plt.close()
# Print classification report
print(classification_report(y_test, model.predict(X_test)))
#roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
# Plot the ROC curve
model_ROC = plot_roc_curve(model, X_test, y_test)
plt.tight_layout()
plt.savefig("roc.png",dpi=120) 
plt.close()

GitHub Workflows and CML: Create a new workflow directory and a new file under it by name cml.yaml. You can name the file anything you want to but ensure the extension is .yaml

Define the process in the yaml as below: 

1. We want to trigger the process every time there is a code push to the repository.
2. We will use the docker image from CML in this case but, we can also use our own docker if required.
3. The dependencies that we had defined in requirements.txt file will be installed on the environment.
4. Our model file train.py will be executed at the end.

name: model-CHD
on: [push]
jobs:
  run:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/cml-py3:latest
    steps:
      - uses: actions/checkout@v2
      - name: 'Train my model'
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Your ML workflow goes here
          pip install -r requirements.txt
          python train.py

We also had generated some metrics like confusion matrix and ROC curve. It would be great to have them displayed in the markdown format which will help us with our experiments and also to measure/track the model’s performance.

echo "## Model Metrics" >> report.md
cat metrics.txt >> report.md
# Write your CML report
echo "## Model Visualization" >> report.md 
cml-publish cm.png --md >> report.md
cml-publish roc.png --md >> report.md
cml-send-comment report.md

Refer to the complete YAML file from the git repository.

Once you commit the YAML file, the workflow will be triggered as shown below.

Expand the Train my model tab and there is our classification report we had planned to print on the console.

Navigate your commits on GitHub and you can see the results and both the plots in markdown format.

Great, we have been able to load the data, build a model, create a workflow, and execute the whole process on git push.

Now, let’s experiment: Once we make any changes to the code and push it to GitHub, the entire workflow will be triggered and a new markdown report will be generated specific to our experiment.

• we will change the test size from 0.2 to 0.25

X_train, X_test, y_train, y_test = train_test_split(df_heart, y, test_size=0.25, random_state=seed)

• We had initially dropped a variable by name famhist, lets dummy it and use it in our model

df_heart = pd.get_dummies(df_heart, columns = ['famhist'], drop_first=True)