Python „Code check” implementation in IDE + Github action
  • Testing
  • Share this post

    Intro 

    Lint tools are something that make our lives, developer lives, easier. These useful components help us be better, faster and more productive. Imagine the situation when you run code that was just written, and it fails… You check traceback, fixing code, run it again, and it fails, again and again. That’s because of stopping on first encountered bug. You can detect a lot of errors with Python code checker tools before you even run your code. However, linters do more than only check code for errors. They also check styling. Everyone knows that keep styling standard is very important in a project. It helps read code, work in the team, add new features and especially find bugs.

    We have a lot of Python lints to choose. One of the most popular is flake8. This is a combination of many tools, but mainly Pyflakes (fast and low rate false positives python error checker app) and Pep8/Pycodestyle (check code against pep8 convention). It is easy to use, and it has a lot of extensions you can use to check your code deeply and set a strict style rules.

    Flake8 basics

    The first step to use flake8 is to install it in your environment.

    Note
    It is good practice to use a virtual environment for every particular application. You can use for example

    venv

    Bash
    python -m pip install flake8

    You can invoke flak8 via command-line or via python.

    Bash
    flake8 path/to/your/file.py
    # or
    python -m flake8 path/to/your/file.py

    This way, you will check Python code using default configuration. But you can simply customize it.

    Bash
    flake8 --max-line-length 120 path/to/your/file.py

    We set max line length to 120 characters instead of 79 according to pep8. You can find full list of options here.

    In most cases we use more complex configuration that we want to reuse. Flake8 allows us to store preferences in configuration file. Example configuration file might look like this:

    txt
    [flake8]
    ignore = D203
    exclude =
        .git,
        __pycache__,
        docs/source/conf.py,
        old,
        build,
        dist
    max-complexity = 10
    

    Enable linter in your IDE

    The use of flake8 from command line works, but better is using it directly from your preferred Python IDE! We can see code errors and style inaccuracy in fly. Moreover, IDE will highlight faulty line/object. In this article we will show how to enable python liner in VCS IDE, but you can enable it in your favorite tool.

    First of all, make sure that you have installed flake8 in your environment. Next, open Virtual Code Studio, press F1 and start typing “Python: Select linter”.

    And choose flake8 from the list.

    That is it. Now we can see output from flake in “problems” tab in IDE.

    As we mentioned before, there are a lot of extensions that we can use to adjust flake8 to our requirements. For example, imagine that in our project we would like to have the Pydoc documentation in order to PEP 257 rules. We can use flake8-docstrings extension. For that purpose install package using pip:

    Bash
    python -m pip install flake8-docstrings

    Reload flake8 check by save file (Ctrl + s) and see what we get

    As we mentioned before, there are a lot of extensions that we can use to adjust flake8 to our requirements. For example, imagine that in our project we would like to have the Pydoc documentation in order to PEP 257 rules. We can use flake8-docstrings extension. For that purpose install package using pip:

    Bash
    python -m pip install flake8-docstrings

    Reload flake8 check by save file (Ctrl + s) and see what we get

    We can also attach configuration file with all setting adjusted to our requirements while using flake8 from IDE. Let’s assume that we don’t need to check D403 – “First word of the first line should be properly capitalized”. The configuration file should look like this:

    txt
    [flake8]
    ignore = D403

    Now we need to configure VCS to use this file. Add new line in settings.json file under .vscode subfolder 

    Json
    {"python.linting.flake8Args":  ["--append-config=/path/to/flake8/config/.flake8"]}

    The file should look like this:

    Perfect! The flake8 doesn’t check D403 anymore.

    “Check code” task in GitHub actions

    So far we have checked our code locally, but we can do the same automatically. Code check/review is a part of CI/CD (CI – Continues Integration to be specifically). This way we can check code with every push to repository, with PR (pull request) or whatever you wish, depending on trigger you will set. When we use GitHub to keep our code in a repository, there is an easy way to implement it – “GitHub actions”. “GitHub actions” is a built-in CI/CD tool. Even in the “free” account plan in this most popular git repository platform offers 2,000 minutes per month for automation. We can use them to build, test and deploy our code in the way we want.

    First of all, let’s create a repository. You can do it here.

    You can create your own practice. Let’s initiate git repository and push their files. Run commands in terminal

    Bash
    git init
    git add flake8_demo.py, .flake8
    git commit -m “init commit”
    git remote add origin [email protected]:user_name/your_repo_name.git
    git push origin master
    Note
    I’m using ssh connection with ssh keys. You can find more information about it here. But you can use https as well.

    We are ready to create GitHub action workflow. We need to create additional directory structure under our repository. Path .github/workflows will be recognized by GitHub and declared workflows will be processed. “Actions” are written in YAML. 

    This is how “Python Checker” action can be setup.

    Yaml
    name: PythonChecker
    
    on:
      pull_request:
        branches: [ master ]
    
    jobs:
      flake8check:
        runs-on: ubuntu-latest
        strategy:
          matrix:
            python-version: [ 3.8 ]
    
      name: flake8 check (via Python${{ matrix.python-version }})
      steps:
        - name: Checkout code repo
          uses: actions/checkout@v2
    
        - name: Set up Python${{ matrix.python-version }}
          uses: actions/setup-python@v2
          with:
            python-version: ${{ matrix.python-version }}
    
          - name: Install flake8 extensions
            run: |
            python -m pip install --upgrade pip
            if [ -f ./requirements-flake8.txt ]; then python -m pip install -r ./requirements-flake8.txt; fi
    
         - name: Run check
           run: |
    
           #stop the check if there are flake8 errors
           flake8  --config=.flake8

    Let’s try to understand how it works. In the first line we declare name of our action. This name will be visible in the “Action” tab of the portal – I’ll show that later.

    In the second main node, we specify a trigger for our workflow. Our action will start at PR to master branch.

    Under “jobs” node, we describe environment and particular steps of workflow. Every workflow can have many jobs. In our case there is one job ID “flake8check” which will be run on latest version of ubuntu OS with matrix list variable “python-version” with one value ‘3.8’. Matrix feature can be used to run the same job, for example with different version of pythons. Our job will be named “flake8 check (via Python {matrix_variable})” and this name will be displayed in GitHub. Finally, we came to a description of the steps.

    There are four steps with name parameter and step configuration:

    • “Checkout code repo” pull branch code which we are want to merge with “master” using “actions/checkout@v2”. This is action from Github action marketplace.
    • “Set up Python” is another action from marketplace which is installation of python on our environment.
    • “Install flak8 extensions” use “run” command to execute provided code in OS terminal. It is upgrading pip tool and installation of all necessary python modules (flake8 and flake8-docstrings)
    • The last “Run check” step is calling flake8 application with our configuration file.

    Now we know the code, so let’s check how it works.

    Bash
    git checkout -b python_checker
    git add .github/workflows/python_checker.yaml requirements-flake8.txt
    git commit -m “add GitHub action”

    I created PR from “python_checker” branch to “master” on Github portal, because it triggers our action. After that, we can see results under “Actions” tab.

    As we suspect job failed. We can check details after clicking in workflow run.

    GitHub’s actions are easy to develop, and we can use it in many other tasks. Moreover, even in free account plan we get 2000 minutes for our jobs. All code used in this article you can find on https://github.com/michalmi-dss/flake8_check_code

     

    Check out our blog for more details on Data Engineering:

     

    Data Engineering

    Share this post
    Close

    Send Feedback