FWIW.CO.ZA

Blog of Christiaan Rademan.

How To Package Your Python Project

I've recently started working on new open-source projects and decided to publish a post a basic tutorial on how to package your Python code.

This tutorial doesn't describe the only way of doing things, merely one specific approach that I use.

Package/Module Names

Python module/package names should generally follow the following constraints:

  • All lowercase
  • Unique on pypi, even if you don’t want to make your package publicly available (you might want to specify it privately as a dependency later)
  • Underscore-separated or no word separators at all (don’t use hyphens)

File Structure

The structure below

Example:

myproject/
    myproject/
        __init__.py
        a_module.py
    setup.py
    setup.cfg
    docs/
    tests/
    Authors
    README.rst
    LICENSE
    ChangeLog
    MANIFEST.in
    requirements.txt
  • myproject/myproject contains your package with modules.

requirements.txt

"Requirements files" are files containing a list of items to be installed using pip install like so:

$ pip install -r requirements.txt

In our case we use the requirements.txt file in the setup.py with setuptools to install dependencies.

MANIFEST.in

The myproject/MANIFEST.in includes additional files to the package.

include Authors
include README.rst
include LICENSE
include requirements.txt

setup.py

import os

try:
    from setuptools import setup
    from setuptools import find_packages
except Exception as e:
    print("Requires 'setuptools'")
    print(" pip install setuptools")
    exit()

config = {
    "name": "myproject",
    "version": "0.0.0",
    "author": "Christiaan F Rademan",
    "author_email": "chris@fwiw.co.za",
    "description": "MyProject Package Example",
    "license": "BSD 3-Clause",
    "keywords": "another example",
    "url": "http://www.fwiw.co.za",
    "packages": find_packages(),
    "include_package_data": True,
    "classifiers": [
        "Topic :: Software Development :: Libraries :: Application Frameworks",
        "Environment :: Other Environment",
        "Intended Audience :: Information Technology",
        "Intended Audience :: System Administrators",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: BSD License",
        "Operating System :: POSIX :: Linux",
        "Programming Language :: Python",
        "Programming Language :: Python :: 2.7"
        ]
    }

# allow setup.py to be run from any path
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))

if os.path.exists(os.path.join(os.path.dirname(__file__),
                               'requirements.txt')):
    with open(os.path.join(os.path.dirname(__file__),
                           'requirements.txt')) as x:
        requirements = x.read().splitlines()
else:
    requirements = []

with open(os.path.join(os.path.dirname(__file__), 'README.rst')) as x:
    readme = x.read()

print("%s %s\n" % (config['name']))

setup(
    install_requires=requirements,
    long_description=readme,
    **config
)

You can use setup.py to register your project on PyPI which is explained later or use it to remove, install your package.

$ python setup.py install

will install the package

$ python setup.py develop

The develop will not install the package but it will create a .egg-link in the deployment directory back to the project source code directory.

So it's like installing but instead of copying to the site-packages it adds a symbolic link (the .egg-link acts as a multiplatform symbolic link).

That way you can edit the source code and see the changes directly without having to reinstall every time that you make a little change. This is useful when you are the developer of that project hence the name develop.

setup.cfg

This tells PyPI where your README file is.

[metadata]
description-file = README.rst

LICENSE.txt

This file will contain whichver license you want your code to have. I tend to use the BSD 3-Clause license.

What is PyPI?

From the official website:

PyPI — the Python Package Index

The Python Package Index is a repository of software for the Python programming language.

Upload your code on PyPI. It's a big list of python packages that you absolutely must submit your package to for it to be easily one-line installable.

PyPI Account

On PyPI Live and also on PyPI Test, you must create an account in order to be able to upload your code. I recommend using the same email/password for both accounts, just to make your life easier when it comes time to push.

~.pypirc configuration file

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=your_username

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username

Upload your package to PyPI Test

This will attempt to register your package against PyPI's test server, just to make sure you've set up everything correctly.

$ python setup.py register -r pypitest

Now upload your package

$ python setup.py sdist upload -r pypitest

Upload your package to PyPI Test

Simply run above commands again and replace pypitest with pypi

Error while Uploading

If you receive an error TypeError: cannot concatenate 'str' and 'NoneType' objects this is a known bug. Its because the upload is not prompting for the password.

To work around this register and upload at the same time like so:

$ python setup.py sdist register -r pypi upload -r pypi

VIM, Plugins and Python ;-)

VI Improved (VIM) is my editor of choice. I prefer to use this as development and scripting IDE. This is also seems to be the general view of the open-source community. VIM is ubiquitous, fast, and never crashes. And it can do just about anything!

Installing VIM

You require vim compiled with Python support.

Ubuntu

apt-get install vim-nox-py2

OSX

BREW

$ brew update
$ brew install vim

MACPORTS

$ sudo port update
$ sudo port install vim +huge +python27

VIM Extensions

VIM has several extension managers, but the one I strongly recommend is Vundle. Think of it as pip for VIM. It makes installing and updating packages trivial.

Install Vundle

$ git clone https://github.com/gmarik/Vundle.vim.git ~/.vim/bundle/Vundle.vim

This command downloads the Vundle plugin manager and chucks it in your VIM bundles directory. Now you can manage all your extensions from the .vimrc configuration file.

Add the file to your user’s home directory:

$ touch ~/.vimrc

Now set up Vundle in your .vimrc and some extra plugins using:

set nocompatible " be iMproved, required
filetype off     " required

" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

" let Vundle manage Vundle, required
Plugin 'gmarik/Vundle.vim'

" Add all your plugins here (note older versions of Vundle used Bundle instead of Plugin)
Plugin 'tmhedberg/SimpylFold'
Plugin 'vim-scripts/indentpython.vim'
Bundle 'Valloric/YouCompleteMe'
Plugin 'scrooloose/syntastic'
Plugin 'nvie/vim-flake8'
Plugin 'jnurmine/Zenburn'
Plugin 'altercation/vim-colors-solarized'
Plugin 'Lokaltog/powerline', {'rtp': 'powerline/bindings/vim/'}

" All of your Plugins must be added before the following line
call vundle#end()            " required
filetype plugin indent on    " required

" Enable folding
set foldmethod=indent
set foldlevel=99
" Enable folding with the spacebar
nnoremap <space> za

" YCM
map <leader>g  :YcmCompleter GoToDefinitionElseDeclaration<CR>
let g:ycm_autoclose_preview_window_after_insertion=1
let g:ycm_add_preview_to_completeopt=1

set encoding=utf-8

syntax on
set tabstop=4
set softtabstop=4
set shiftwidth=4
set expandtab
set autoindent
set fileformat=unix
autocmd FileType python set textwidth=79
autocmd FileType sh nnoremap <F5> :!%:p<CR>
autocmd FileType php nnoremap <F5> :!clear & php %:p<CR>
autocmd FileType python nnoremap <F5> :!clear & python %:p<CR>
autocmd FileType python nnoremap <F5> :!clear & python %:p<CR>
autocmd FileType python nnoremap <F6> :!clear & pudb %:p<CR>
autocmd FileType python map <buffer> <F4> :call Flake8()<CR>
autocmd FileType python imap <F3> <Esc>:YcmCompleter GetDoc<CR>i

set nu
set cursorline
highlight LineNr ctermbg=white ctermfg=black
highlight CursorLineNr ctermbg=darkred ctermfg=black
highlight ExtraWhitespace ctermbg=red guibg=red
match ExtraWhitespace /\s\+$/
autocmd BufWinEnter * match ExtraWhitespace /\s\+$/
autocmd InsertEnter * match ExtraWhitespace /\s\+\%#\@<!$/
autocmd InsertLeave * match ExtraWhitespace /\s\+$/
autocmd BufWinLeave * call clearmatches()

set laststatus=2
hi statusline ctermbg=black ctermfg=grey

let python_highlight_all = 1

if has('gui_running')
    set background=dark
    colorscheme solarized
else
    colorscheme zenburn
endif

"python with virtualenv support
py << EOF
import os
import sys
if 'VIRTUAL_ENV' in os.environ:
    project_base_dir = os.environ['VIRTUAL_ENV']
    activate_this = os.path.join(project_base_dir, 'bin/activate_this.py')
    execfile(activate_this, dict(__file__=activate_this))
EOF

Fire up VIM and run to install all the plugins:

:PluginInstall

This command tells Vundle to work its magic – downloading all the plugins and installing/updating them for you.

Install YCM - Important for autocomplete

$ cd ~/.vim/bundle/YouCompleteMe
$ ./install.sh

That’s it. You’re now set up!!

Plugins

Short introductions to above plugins installed...

Simplyfold

Plugin 'tmhedberg/SimpylFold'

Most “modern” IDEs provide a way to collapse (or fold) methods and classes, showing you just the class/method definition lines instead of all the code. Use <space bar> or za to hide or unhide class cursor is on.

Auto-Indention

Plugin 'vim-scripts/indentpython.vim'

Autoindent will help but in some cases (like when a function signature spans multiple lines), it doesn’t always do what you want, especially when it comes to conforming to PEP8 standards. To fix that, we can use the indentpython.vim extension

Auto-complete

Bundle 'Valloric/YouCompleteMe'

Under the hood YouCompleteMe uses a few different auto-completers (including Jedi for Python), and it needs some C libraries to be installed for it to work correctly. The docs have very good installation instructions so I won’t repeat them here, but be sure you follow them.

Syntax Checking/Highlighting

Plugin 'scrooloose/syntastic'

Plugin 'nvie/vim-flake8'

You can have VIM check your syntax on each save with the syntastic extension including PEP8 checking.

Color Schemes

Plugin 'jnurmine/Zenburn'

Plugin 'altercation/vim-colors-solarized'

Color schemes work in conjunction with the basic color scheme that you are using. Check out solarized for GUI mode, and Zenburn for terminal mode

Powerline

Plugin 'Lokaltog/powerline', {'rtp': 'powerline/bindings/vim/'}

Powerline is a status bar that displays things like the current virtualenv, git branch, files being edited, and much more.

It’s written in Python, and it supports a number of other environments like zsh, bash, tmux, and IPython.

GIT 101

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

Based on this article I am using GITHUB http://www.github.com for hosting my repositories securely online. Code is hosted on GITHUB makes the code easy to browse, download, fork, etc.

Clone a project

Cloning a project is creating a local copy on your host of a remote repository.

$ git clone https://github.com/vision1983/nfw.git

Joining / Fork a project

On GITHUB you can create a fork which is a copy of repository hosted on GITHUB within your own GITHUB space.

Once you pushed your changes to own copy on GITHUB you can do a pull request on GITHUB which will inform the maintainers of the original project that you wish to merge your changes upstream.

Creating your own GITHUB project

Create a repository on GITHUB without initializing.

  • Enter your source code directory
  • Create a new repository with empty README.md or README.rst.
$ git init
$ git add README.md
$ git commit -m "first commit"
$ git remote add origin https://github.com/vision1983/test.git
$ git push -u origin master

or push exisiting repository to GITHUB

git remote add origin https://github.com/vision1983/test.git
git push -u origin master

Working with local repository

To commit changes you need to stage the changes using the following commands:

Add file to repository

$ git add file

Remove file from repository

$ git del file

Before committing you can undo or checkout original file

$ git checkout file

Commit changes locally

$ git commit -a

Finally upload or push to GITHUB

$ git push

Syncing a fork

Sync a fork of a repository to keep it up-to-date with the upstream repository.

Before you can sync your fork with an upstream repository, you must configure a remote that points to the upstream repository in GITHUB.

List current configured remote repository for your fork

$ git remote -v
origin  https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
origin  https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)

Specify a new remote upstream repository that will be synced with the fork

$ git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.

Verify the new upstream repository you've specified for your fork

$ git remote -v
origin https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
origin https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)
upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (fetch)
upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (push)

Fetch the branches and their respective commits from the upstream repository. Commits to master will be stored in a local branch, upstream/master

$ git fetch upstream
remote: Counting objects: 75, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 62 (delta 27), reused 44 (delta 9)
Unpacking objects: 100% (62/62), done.
From https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY
* [new branch]      master     -> upstream/master

Check out your fork's local master branch

$ git checkout master
Switched to branch 'master'

Merge the changes from upstream/master into your local master branch. This brings your fork's master branch into sync with the upstream repository, without losing your local changes

$ git merge upstream/master

GIT Tags

Git has the ability to tag specific points in history as being important. Typically people use this functionality to mark release points (1.0.0, and so on)

List your tags

$ git tag
0.0.1
0.0.2

Git uses two main types of tags

  • lightweight
  • and annotated.

Annotated Tags

Annotated as per GIT Documentation

Annotated tags, however, are stored as full objects in the Git database. They’re checksummed; contain the tagger name, email, and date; have a tagging message; and can be signed and verified with GNU Privacy Guard (GPG). It’s generally recommended that you create annotated tags so you can have all this information; but if you want a temporary tag or for some reason don’t want to keep the other information, lightweight tags are available too.

Creating an annotated tag

The easiest way is to specify -a when you run the tag command.

$ git tag -a 0.0.2 -m "my version 0.0.2"
$ git tag
0.0.0
0.0.1
0.0.2

The -m specifies a tagging message, which is stored with the tag.

You can see the tag data along with the commit that was tagged by using the git show command

$ git show 0.0.2

Lightweight Tags

Lightweight as per GIT Documentation

A lightweight tag is very much like a branch that doesn’t change. It just points to a specific commit.

To create a lightweight tag, don’t supply the -a, -s, or -m option

$ git tag 0.0.3
$ git tag
0.0.0
0.0.1
0.0.2
0.0.3

When you run git show on the tag, you don’t see the extra tag information as per with annotated tag.

Tagging Later

You can also tag commits after you’ve moved past them.

Retrieve commit history looks like this

$ git log --pretty=oneline

Tag based on checksum from history

$ git tag -a 0.0.6 [checksum]

Sharing Tags

By default, the git push command doesn’t transfer tags to remote servers. You will have to explicitly push tags to a shared server after you have created them.

$ git push origin [tagname]

If you have a lot of tags that you want to push up at once, you can

$ git push origin --tags

Delete Tags

Delete tag locally

$ git tag -d 0.0.1

Delete tag from upstream

$ git push origin :refs/tags/1.0.5

Now, when someone else clones or pulls from your repository, they will get all your tags as well.

Managing Branches

In your GITHUB project, you need to keep your master branch clean, by clean I mean without any changes, like that you can create at any time a branch from your master. Each time, that you want to commit a bug or a feature, you need to create a branch for it, which will be a copy of your master branch.

When you do a pull request on a branch, you can continue to work on another branch and make another pull request on this other branch.

Before creating a new branch, pull the changes from upstream. Your master needs to be up to date.

Create the branch on your local machine and switch in this branch

$ git checkout -b [name_of_your_new_branch]

Push the branch on github

$ git push origin [name_of_your_new_branch]

When you want to commit something in your branch, be sure to be in your branch.

You can see all branches created by using

$ git branch

See details on branches for origin/GITHUB

$ git remote show origin

Switch to a branch

$ git checkout [name_of_your_new_branch]

Pull updates from branch on GITHUB

$ git pull origin [name_of_your_new_branch]

Push updates to your branch on GITHUB

$ git push origin [name_of_your_new_branch]

Delete a branch on your local filesystem

$ git branch -d [name_of_your_new_branch]

Force the deletion of local branch on your filesystem

$ git branch -D [name_of_your_new_branch]

Delete the branch on github

$ git push origin :[name_of_your_new_branch]

Python IP Calculator module

pyipcalc is module that I developed for doing simple ip calculations needed by some python applications. Python 3 has built-in 'ipaddress' module providing many of the desired functionality. However I needed a common interface and support within both Python 2.7 and 3.x. At the time there were many modules some which were complex and others that were broken at the time. pyipcalc is simple and contributions are welcome!

PYIPCALC Website.