FWIW.CO.ZA

Blog of Christiaan Rademan.

How To Package Your Python Project

I've recently started working on new open-source projects and decided to publish a post a basic tutorial on how to package your Python code.

This tutorial doesn't describe the only way of doing things, merely one specific approach that I use.

Package/Module Names

Python module/package names should generally follow the following constraints:

  • All lowercase
  • Unique on pypi, even if you don’t want to make your package publicly available (you might want to specify it privately as a dependency later)
  • Underscore-separated or no word separators at all (don’t use hyphens)

File Structure

The structure below

Example:

myproject/
    myproject/
        __init__.py
        a_module.py
    setup.py
    setup.cfg
    docs/
    tests/
    Authors
    README.rst
    LICENSE
    ChangeLog
    MANIFEST.in
    requirements.txt
  • myproject/myproject contains your package with modules.

requirements.txt

"Requirements files" are files containing a list of items to be installed using pip install like so:

$ pip install -r requirements.txt

In our case we use the requirements.txt file in the setup.py with setuptools to install dependencies.

MANIFEST.in

The myproject/MANIFEST.in includes additional files to the package.

include Authors
include README.rst
include LICENSE
include requirements.txt

setup.py

import os

try:
    from setuptools import setup
    from setuptools import find_packages
except Exception as e:
    print("Requires 'setuptools'")
    print(" pip install setuptools")
    exit()

config = {
    "name": "myproject",
    "version": "0.0.0",
    "author": "Christiaan F Rademan",
    "author_email": "chris@fwiw.co.za",
    "description": "MyProject Package Example",
    "license": "BSD 3-Clause",
    "keywords": "another example",
    "url": "http://www.fwiw.co.za",
    "packages": find_packages(),
    "include_package_data": True,
    "classifiers": [
        "Topic :: Software Development :: Libraries :: Application Frameworks",
        "Environment :: Other Environment",
        "Intended Audience :: Information Technology",
        "Intended Audience :: System Administrators",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: BSD License",
        "Operating System :: POSIX :: Linux",
        "Programming Language :: Python",
        "Programming Language :: Python :: 2.7"
        ]
    }

# allow setup.py to be run from any path
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))

if os.path.exists(os.path.join(os.path.dirname(__file__),
                               'requirements.txt')):
    with open(os.path.join(os.path.dirname(__file__),
                           'requirements.txt')) as x:
        requirements = x.read().splitlines()
else:
    requirements = []

with open(os.path.join(os.path.dirname(__file__), 'README.rst')) as x:
    readme = x.read()

print("%s %s\n" % (config['name']))

setup(
    install_requires=requirements,
    long_description=readme,
    **config
)

You can use setup.py to register your project on PyPI which is explained later or use it to remove, install your package.

$ python setup.py install

will install the package

$ python setup.py develop

The develop will not install the package but it will create a .egg-link in the deployment directory back to the project source code directory.

So it's like installing but instead of copying to the site-packages it adds a symbolic link (the .egg-link acts as a multiplatform symbolic link).

That way you can edit the source code and see the changes directly without having to reinstall every time that you make a little change. This is useful when you are the developer of that project hence the name develop.

setup.cfg

This tells PyPI where your README file is.

[metadata]
description-file = README.rst

LICENSE.txt

This file will contain whichver license you want your code to have. I tend to use the BSD 3-Clause license.

What is PyPI?

From the official website:

PyPI — the Python Package Index

The Python Package Index is a repository of software for the Python programming language.

Upload your code on PyPI. It's a big list of python packages that you absolutely must submit your package to for it to be easily one-line installable.

PyPI Account

On PyPI Live and also on PyPI Test, you must create an account in order to be able to upload your code. I recommend using the same email/password for both accounts, just to make your life easier when it comes time to push.

~.pypirc configuration file

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=your_username

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username

Upload your package to PyPI Test

This will attempt to register your package against PyPI's test server, just to make sure you've set up everything correctly.

$ python setup.py register -r pypitest

Now upload your package

$ python setup.py sdist upload -r pypitest

Upload your package to PyPI Test

Simply run above commands again and replace pypitest with pypi

Error while Uploading

If you receive an error TypeError: cannot concatenate 'str' and 'NoneType' objects this is a known bug. Its because the upload is not prompting for the password.

To work around this register and upload at the same time like so:

$ python setup.py sdist register -r pypi upload -r pypi

VIM, Plugins and Python ;-)

VI Improved (VIM) is my editor of choice. I prefer to use this as development and scripting IDE. This is also seems to be the general view of the open-source community. VIM is ubiquitous, fast, and never crashes. And it can do just about anything!

Installing VIM

You require vim compiled with Python support.

Ubuntu

apt-get install vim-nox-py2

OSX

BREW

$ brew update
$ brew install vim

MACPORTS

$ sudo port update
$ sudo port install vim +huge +python27

VIM Extensions

VIM has several extension managers, but the one I strongly recommend is Vundle. Think of it as pip for VIM. It makes installing and updating packages trivial.

Install Vundle

$ git clone https://github.com/gmarik/Vundle.vim.git ~/.vim/bundle/Vundle.vim

This command downloads the Vundle plugin manager and chucks it in your VIM bundles directory. Now you can manage all your extensions from the .vimrc configuration file.

Add the file to your user’s home directory:

$ touch ~/.vimrc

Now set up Vundle in your .vimrc and some extra plugins using:

set nocompatible " be iMproved, required
filetype off     " required

" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

" let Vundle manage Vundle, required
Plugin 'gmarik/Vundle.vim'

" Add all your plugins here (note older versions of Vundle used Bundle instead of Plugin)
Plugin 'tmhedberg/SimpylFold'
Plugin 'vim-scripts/indentpython.vim'
Bundle 'Valloric/YouCompleteMe'
Plugin 'scrooloose/syntastic'
Plugin 'nvie/vim-flake8'
Plugin 'jnurmine/Zenburn'
Plugin 'altercation/vim-colors-solarized'
Plugin 'Lokaltog/powerline', {'rtp': 'powerline/bindings/vim/'}

" All of your Plugins must be added before the following line
call vundle#end()            " required
filetype plugin indent on    " required

" Enable folding
set foldmethod=indent
set foldlevel=99
" Enable folding with the spacebar
nnoremap <space> za

" YCM
map <leader>g  :YcmCompleter GoToDefinitionElseDeclaration<CR>
let g:ycm_autoclose_preview_window_after_insertion=1
let g:ycm_add_preview_to_completeopt=1

set encoding=utf-8

syntax on
set tabstop=4
set softtabstop=4
set shiftwidth=4
set expandtab
set autoindent
set fileformat=unix
autocmd FileType python set textwidth=79
autocmd FileType sh nnoremap <F5> :!%:p<CR>
autocmd FileType php nnoremap <F5> :!clear & php %:p<CR>
autocmd FileType python nnoremap <F5> :!clear & python %:p<CR>
autocmd FileType python nnoremap <F5> :!clear & python %:p<CR>
autocmd FileType python nnoremap <F6> :!clear & pudb %:p<CR>
autocmd FileType python map <buffer> <F4> :call Flake8()<CR>
autocmd FileType python imap <F3> <Esc>:YcmCompleter GetDoc<CR>i

set nu
set cursorline
highlight LineNr ctermbg=white ctermfg=black
highlight CursorLineNr ctermbg=darkred ctermfg=black
highlight ExtraWhitespace ctermbg=red guibg=red
match ExtraWhitespace /\s\+$/
autocmd BufWinEnter * match ExtraWhitespace /\s\+$/
autocmd InsertEnter * match ExtraWhitespace /\s\+\%#\@<!$/
autocmd InsertLeave * match ExtraWhitespace /\s\+$/
autocmd BufWinLeave * call clearmatches()

set laststatus=2
hi statusline ctermbg=black ctermfg=grey

let python_highlight_all = 1

if has('gui_running')
    set background=dark
    colorscheme solarized
else
    colorscheme zenburn
endif

"python with virtualenv support
py << EOF
import os
import sys
if 'VIRTUAL_ENV' in os.environ:
    project_base_dir = os.environ['VIRTUAL_ENV']
    activate_this = os.path.join(project_base_dir, 'bin/activate_this.py')
    execfile(activate_this, dict(__file__=activate_this))
EOF

Fire up VIM and run to install all the plugins:

:PluginInstall

This command tells Vundle to work its magic – downloading all the plugins and installing/updating them for you.

Install YCM - Important for autocomplete

$ cd ~/.vim/bundle/YouCompleteMe
$ ./install.sh

That’s it. You’re now set up!!

Plugins

Short introductions to above plugins installed...

Simplyfold

Plugin 'tmhedberg/SimpylFold'

Most “modern” IDEs provide a way to collapse (or fold) methods and classes, showing you just the class/method definition lines instead of all the code. Use <space bar> or za to hide or unhide class cursor is on.

Auto-Indention

Plugin 'vim-scripts/indentpython.vim'

Autoindent will help but in some cases (like when a function signature spans multiple lines), it doesn’t always do what you want, especially when it comes to conforming to PEP8 standards. To fix that, we can use the indentpython.vim extension

Auto-complete

Bundle 'Valloric/YouCompleteMe'

Under the hood YouCompleteMe uses a few different auto-completers (including Jedi for Python), and it needs some C libraries to be installed for it to work correctly. The docs have very good installation instructions so I won’t repeat them here, but be sure you follow them.

Syntax Checking/Highlighting

Plugin 'scrooloose/syntastic'

Plugin 'nvie/vim-flake8'

You can have VIM check your syntax on each save with the syntastic extension including PEP8 checking.

Color Schemes

Plugin 'jnurmine/Zenburn'

Plugin 'altercation/vim-colors-solarized'

Color schemes work in conjunction with the basic color scheme that you are using. Check out solarized for GUI mode, and Zenburn for terminal mode

Powerline

Plugin 'Lokaltog/powerline', {'rtp': 'powerline/bindings/vim/'}

Powerline is a status bar that displays things like the current virtualenv, git branch, files being edited, and much more.

It’s written in Python, and it supports a number of other environments like zsh, bash, tmux, and IPython.

Python IP Calculator module

pyipcalc is module that I developed for doing simple ip calculations needed by some python applications. Python 3 has built-in 'ipaddress' module providing many of the desired functionality. However I needed a common interface and support within both Python 2.7 and 3.x. At the time there were many modules some which were complex and others that were broken at the time. pyipcalc is simple and contributions are welcome!

PYIPCALC Website.