Python „Code check” implementation in IDE + Github action

Blog > Testing > Python „Code check” implementation in IDE + Github action
Python „Code check” implementation in IDE + Github action
Share this post

Intro 

Lint tools are something that make our lives, developer lives, easier. These useful components help us be better, faster and more productive. Imagine the situation when you run code that was just written, and it fails… You check traceback, fixing code, run it again, and it fails, again and again. That’s because of stopping on first encountered bug. You can detect a lot of errors with Python code checker tools before you even run your code. However, linters do more than only check code for errors. They also check styling. Everyone knows that keep styling standard is very important in a project. It helps read code, work in the team, add new features and especially find bugs.

We have a lot of Python lints to choose. One of the most popular is flake8. This is a combination of many tools, but mainly Pyflakes (fast and low rate false positives python error checker app) and Pep8/Pycodestyle (check code against pep8 convention). It is easy to use, and it has a lot of extensions you can use to check your code deeply and set a strict style rules.

Flake8 basics

The first step to use flake8 is to install it in your environment.

Note
 It is good practice to use a virtual environment for every particular application. You can use for example

venv

Bash
python -m pip install flake8

You can invoke flak8 via command-line or via python.

Bash
flake8 path/to/your/file.py
# or
python -m flake8 path/to/your/file.py

This way, you will check Python code using default configuration. But you can simply customize it.

Bash
flake8 --max-line-length 120 path/to/your/file.py

We set max line length to 120 characters instead of 79 according to pep8. You can find full list of options here.

In most cases we use more complex configuration that we want to reuse. Flake8 allows us to store preferences in configuration file. Example configuration file might look like this:

txt
[flake8]
ignore = D203
exclude =
    .git,
    __pycache__,
    docs/source/conf.py,
    old,
    build,
    dist
max-complexity = 10

Enable linter in your IDE

The use of flake8 from command line works, but better is using it directly from your preferred Python IDE! We can see code errors and style inaccuracy in fly. Moreover, IDE will highlight faulty line/object. In this article we will show how to enable python liner in VCS IDE, but you can enable it in your favorite tool.

First of all, make sure that you have installed flake8 in your environment. Next, open Virtual Code Studio, press F1 and start typing “Python: Select linter”.

And choose flake8 from the list.

That is it. Now we can see output from flake in “problems” tab in IDE.

As we mentioned before, there are a lot of extensions that we can use to adjust flake8 to our requirements. For example, imagine that in our project we would like to have the Pydoc documentation in order to PEP 257 rules. We can use flake8-docstrings extension. For that purpose install package using pip:

Bash
python -m pip install flake8-docstrings

Reload flake8 check by save file (Ctrl + s) and see what we get

As we mentioned before, there are a lot of extensions that we can use to adjust flake8 to our requirements. For example, imagine that in our project we would like to have the Pydoc documentation in order to PEP 257 rules. We can use flake8-docstrings extension. For that purpose install package using pip:

Bash
python -m pip install flake8-docstrings

Reload flake8 check by save file (Ctrl + s) and see what we get

We can also attach configuration file with all setting adjusted to our requirements while using flake8 from IDE. Let’s assume that we don’t need to check D403 – “First word of the first line should be properly capitalized”. The configuration file should look like this:

txt
[flake8]
ignore = D403

Now we need to configure VCS to use this file. Add new line in settings.json file under .vscode subfolder 

Json
{"python.linting.flake8Args":  ["--append-config=/path/to/flake8/config/.flake8"]}

The file should look like this:

Perfect! The flake8 doesn’t check D403 anymore.

“Check code” task in GitHub actions

So far we have checked our code locally, but we can do the same automatically. Code check/review is a part of CI/CD (CI – Continues Integration to be specifically). This way we can check code with every push to repository, with PR (pull request) or whatever you wish, depending on trigger you will set. When we use GitHub to keep our code in a repository, there is an easy way to implement it – “GitHub actions”. “GitHub actions” is a built-in CI/CD tool. Even in the “free” account plan in this most popular git repository platform offers 2,000 minutes per month for automation. We can use them to build, test and deploy our code in the way we want.

First of all, let’s create a repository. You can do it here.

You can create your own practice. Let’s initiate git repository and push their files. Run commands in terminal

Bash
git init
git add flake8_demo.py, .flake8
git commit -m “init commit”
git remote add origin [email protected]:user_name/your_repo_name.git
git push origin master
Note
I’m using ssh connection with ssh keys. You can find more information about it here. But you can use https as well.

We are ready to create GitHub action workflow. We need to create additional directory structure under our repository. Path .github/workflows will be recognized by GitHub and declared workflows will be processed. “Actions” are written in YAML. 

This is how “Python Checker” action can be setup.

Yaml
name: PythonChecker

on:
  pull_request:
    branches: [ master ]

jobs:
  flake8check:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [ 3.8 ]

  name: flake8 check (via Python${{ matrix.python-version }})
  steps:
    - name: Checkout code repo
      uses: actions/checkout@v2

    - name: Set up Python${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

      - name: Install flake8 extensions
        run: |
        python -m pip install --upgrade pip
        if [ -f ./requirements-flake8.txt ]; then python -m pip install -r ./requirements-flake8.txt; fi

     - name: Run check
       run: |

       #stop the check if there are flake8 errors
       flake8  --config=.flake8

Let’s try to understand how it works. In the first line we declare name of our action. This name will be visible in the “Action” tab of the portal – I’ll show that later.

In the second main node, we specify a trigger for our workflow. Our action will start at PR to master branch.

Under “jobs” node, we describe environment and particular steps of workflow. Every workflow can have many jobs. In our case there is one job ID “flake8check” which will be run on latest version of ubuntu OS with matrix list variable “python-version” with one value ‘3.8’. Matrix feature can be used to run the same job, for example with different version of pythons. Our job will be named “flake8 check (via Python {matrix_variable})” and this name will be displayed in GitHub. Finally, we came to a description of the steps.

There are four steps with name parameter and step configuration:

  • “Checkout code repo” pull branch code which we are want to merge with “master” using “actions/checkout@v2”. This is action from Github action marketplace.
  • “Set up Python” is another action from marketplace which is installation of python on our environment.
  • “Install flak8 extensions” use “run” command to execute provided code in OS terminal. It is upgrading pip tool and installation of all necessary python modules (flake8 and flake8-docstrings)
  • The last “Run check” step is calling flake8 application with our configuration file.

Now we know the code, so let’s check how it works.

Bash
git checkout -b python_checker
git add .github/workflows/python_checker.yaml requirements-flake8.txt
git commit -m “add GitHub action”

I created PR from “python_checker” branch to “master” on Github portal, because it triggers our action. After that, we can see results under “Actions” tab.

As we suspect job failed. We can check details after clicking in workflow run.

GitHub’s actions are easy to develop, and we can use it in many other tasks. Moreover, even in free account plan we get 2000 minutes for our jobs. All code used in this article you can find on https://github.com/michalmi-dss/flake8_check_code

 

Check out our blog for more details on Data Engineering:

 

Data Engineering

Author

Share this post