Supercharging automation in terminal with python

Photo by Sai Kiran Anagani on Unsplash

Article overview

Ever since this covid pandemic, I’ve been working and studying remotely. And it is unfortunate of course, but I’m just way more productive than I ever was. And this is the best moment for me to try even harder to upsample my progress. With this in mind, I decided to follow Input less, Output more tact. The very first step that naturally comes to my mind is to include some kind of automation in my day to day workspace. And with that said, let’s get started.

My two cents on automation

Automation is not about just automating

In programming at least, the idea behind any automation is to reduce the effort you put to do any task, without compromising the output of the task. But the reason you should perform automation as much as possible should not be limited to the fact that you wouldn’t have to sit and do the stuff, but that it brings a whole lot of cohesiveness to the code. And, no need to mention that it could save a whole lot of your time depending on the task.

More on cohesive code

This year I have started doing just a lot of projects in Web Dev and ML, (thanks to covid, cough). Thus, I realised, that 20% of all I code every weak, I just the same code like same as in word by word, lol I know. Not only that but my whole workflow from initiating a project to deploying it (in web dev specially) is same too, as in command by command.

So automating these tasks would mean

• Better code quality, hence efficient and fast runtimes.
• Faster debugging, hence more releases.
• More trials, hence more failures and hence, more successes.
• Time-saving, hence more time to focus on core working.
• Less boring, hence more enjoying :)

Automating project initiation

The very first, and the most common task for completing any of my projects, is to start it 😂.

So this definitely and positively should be my first step to automat.

Alright then, now this might sound silly to you but let me lay out the steps I do to get all my projects started.

1. Creating a folder in my workspace directory with the name of the project.
2. Set up a virtual environment and activate it.
3. Creating a file for project specific env variables.
4. Initialize a local GitHub repo.
5. Create a readme file.
6. Initialize a remote GitHub repo.
7. Set the remote URL to the local repo.
8. Create a .gitignore file and common file types, plus that env file.
9. Add, commit, and push an initial upload commit.
10. And then finally, open the project folder via vs code.

So yes, these are a lot of steps, in fact, I even used to forget few of them and later causing me a lot of headaches for solving issues with env variables, package dependencies and whatnot.

Now first of all these steps are not like 100% required all the time, I agree, but they are indeed required for more than 80% of the work I do, especially when I work with other people.

Alright then, if I could somehow hammer these steps out to a single terminal command, which indeed sounds too good to be true, then yupp it would be cool, what else can I say.

Getting started with python

Now I’m gonna be using python for obvious reasons, I mean I could create a bash script, which would run required commands locally, and I could send some get/post requests to GitHub to create a remote repo and do blah blah, but why? I mean why would I ever do that when there’s an API made already for python to do so?

And before we move on, if you’re using windows then make sure to install Git BASH, you would only be able to run our final commands there only.

Okay then, let’s have a look at a basic example of creating a very simple terminal command in python. How about a command that shows and a random programming joke :).

First, you need to write your script. Before that, make sure to install pyjokes

$ pip install pyjokes

And here’s our simple script:

import pyjokes
print(pyjokes.get_joke())

Let’s call this joke.py

Running this script gives outputs like this

Why does Waldo only wear stripes? Because he doesn't want to be spotted.

Did you get it? Me neither. Okay so all good here, what’s next?

First add this line to the very top of your python script.

#!/usr/bin/env python

This tells the bash to run our file as a python script. So now our final script should look like this.

#!/usr/bin/env python
import pyjokes
print(pyjokes.get_joke())

Let's name this file

Now, remove .py from the name of this script file.

$ mv joke.py joke

Then make it executable.

$ chmod +x joke

Now let's create a folder in home directory with name bin that will contain all your custom automation commands.

$ mkdir ~/bin

And then export that folder's path to your terminal's profile.

$ echo 'export PATH=$PATH":$HOME/bin"' >> .profile

You need to do above two steps if you're doing this first time only, so for creating more such terminal commands you can skip above two steps.

And that's it :)

Now you can run simply run joke anywhere in your terminal, and every time you'll get a soy random joke.

So I guess you got the idea, and yeah its not that hard. Okay so let's move to automating my project initiation shall we?

Automating my project initiation

First, let's install some dependencies.

$ pip install pyautogui PyGithub

Now I'm not gonna explain each the code line by line, but will provide explanation to main blocks.

Also, create a GitHub access token export it with name GITHUB_ACCESS_TOKEN, and also the your project path to PROJECT_PATH, where all your projects will be created.

Importing required libraries

import os
import sys
import socket
import time
import pyautogui as auto
from github import Github

Initializing global variables

project_path = os.environ.get("PROJECT_PATH")
backup_path = os.environ.get("BACKUP_PATH")
args = sys.argv
SPACE = " " * 5

Some helper functions

def rmspace(name):
	return name.replace(" ", "-")

def git():
	if check_connected("www.google.com"):
		return Github(os.environ.get("GITHUB_ACCESS_TOKEN"))
	print("Github connection failed..... system is not connected to internet")
	sys.exit(1)

def get_repos():
	return [repo.name for repo in git().get_user().get_repos()]

def check_connected(hostname):
    try:
        host = socket.gethostbyname(hostname)
        s = socket.create_connection((host, 80), 2)
        s.close()
        return True
    except:
    	pass
    return False

def convert_bytes(num):
    for x in ['bytes', 'KB', 'MB', 'GB', 'TB']:
        if num < 1024.0:
            return f"{num:.2f} {x}"
        num /= 1024.0

def get_size(file):    
    path = project_path + file
    
    if os.path.isfile(path):
        return convert_bytes(os.path.getsize(path))

    total_size = 0.0
    for dirpath, dirnames, filenames in os.walk(path):
        for f in filenames:
            fp = os.path.join(dirpath, f)
            if not os.path.islink(fp):
            	try:
            		total_size += os.path.getsize(fp)
            	except:
            		pass
    return convert_bytes(total_size)

def spaceprint(name, size):
    space = SIZE - len(name)
    print(name, end = "")
    print(" " * space, size)

Function check_connected returns True if internet connection is available else False. git logins to your GitHub profile. get_size gives us the size of a file in human readable format.

Main Script

Alright, so below is our full main script which automates everything I discussed earlier.

I've also included help command in the script, just in case you decides to copy my code and require some guidance for usage :)

All right so after putting all those code blocks together, let's move on to last steps. I've named this file as project.py, and then followed all above commands I've explained earlier in this article.

Just run, all you'll get it all.

$ project help

Usage of final script

To create a project

project create <project name> <description> <private (True/False) <homepage url>

Example:

$ project create "new project" "this is my description for GitHub repository" "False"

[============= Initiating project: new project =============]

Creating Readme file ....
Creating Environment Variables ....
Creating Virtual Environment ....
Initiating Git Integration ....
Initialized empty Git repository in <path to your project>/.git/
[master (root-commit) b2963d5] Initial Commit
2 files changed, 3 insertions(+)
create mode 100644 .gitignore
create mode 100644 README.md
Adding remote URL
Pushing initial commit ....
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 282 bytes | 282.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0)
To <https://github.com/braindotai/new-project-name.git>
* [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
Opening project

[============= Project initiated successfully =============]

So now this single command does all that tasks I talked about earlier for my project creation.

Now that's marvelous!!

To print names of projects existing in your PROJECT_PATH

$ project list

Example:

$ project list
1. <First project name>
2. <Second project name>
...

To open a project

project open <project name>

To check directories of a project with folders/files sizes

project <project name>

To delete a project

project delete <project name>

Example:

$ project delete my very old project
Project "my very old project" is deleted successfully

To print repository details of the project

project repo <repo name>

Example:

$ project repo BrainBust

=============== BrainBust ===============
Contents:
      .browserslistrc
      .editorconfig
      .firebase
      .firebaserc
      .gitignore
      README.md
      angular.json
      dist
      e2e
      firebase.json
      karma.conf.js
      ngsw-config.json
      package-lock.json
      package.json
      src
      tsconfig.app.json
      tsconfig.json
      tsconfig.spec.json
      tslint.json

Branches:
      master

Issues: 0
Private: True
Starts: 0
Forked: False
Forks: 0
Watchers: 0
Language: SCSS
Owner: braindotai
URL: https://github.com/braindotai/BrainBust
SSH URL: git@github.com:braindotai/BrainBust.git
=========================================

To list all repositories

project repo list

Example:

$ project repo list
1. <First repository>
2. <First repository>
...

To delete a repository

project repo delete <repository name>

Example:

$ project repo delete new project
Repository "new project" is deleted successfully

To create a backup of a project

project backup <project name>

Example:

$ project backup BrainBust
Project "BrainBust" is backuped successfully

I’ve started using this script from last year, and boy am I happy.

A sample output of the command

And since now you know how to integrate any python script in your project initiation, so now you can customize that script according to your need and do wonders with it, yeah wonders I said.

Remember I talked about code cohesiveness? Though the above script does automate yada yada yada… but no cohesiveness yet! Since I’m just basically automating my mouse movements 😁, the actual code that exist in these projects is still written by me.

Well guess what, I’ve created yet another script which does the project creation for Django specifically on top of the above script, and if you are a Django developer, then you know how much of a lengthy task it is to setup the project.

But wait, not just Django project creation, but also the structure of the whole Django project. So in every project, I’ve automated the creation a User app(for user authentication), and Website app(for basic website layouts and UI/UX designs) with all the required edits and changes we make to the settings.py file. And since this setup is same for every project, so now all take use of all those cohessive code paradigms I discussed earlier.

Alrighty then, I hope you have an awesome idea and know exactly what to automate :)

Thanks for reading.