FWIW.CO.ZA

Blog of Christiaan Rademan.

How To Package Your Python Project

I've recently started working on new open-source projects and decided to publish a post a basic tutorial on how to package your Python code.

This tutorial doesn't describe the only way of doing things, merely one specific approach that I use.

Package/Module Names

Python module/package names should generally follow the following constraints:

  • All lowercase
  • Unique on pypi, even if you don’t want to make your package publicly available (you might want to specify it privately as a dependency later)
  • Underscore-separated or no word separators at all (don’t use hyphens)

File Structure

The structure below

Example:

myproject/
    myproject/
        __init__.py
        a_module.py
    setup.py
    setup.cfg
    docs/
    tests/
    Authors
    README.rst
    LICENSE
    ChangeLog
    MANIFEST.in
    requirements.txt
  • myproject/myproject contains your package with modules.

requirements.txt

"Requirements files" are files containing a list of items to be installed using pip install like so:

$ pip install -r requirements.txt

In our case we use the requirements.txt file in the setup.py with setuptools to install dependencies.

MANIFEST.in

The myproject/MANIFEST.in includes additional files to the package.

include Authors
include README.rst
include LICENSE
include requirements.txt

setup.py

import os

try:
    from setuptools import setup
    from setuptools import find_packages
except Exception as e:
    print("Requires 'setuptools'")
    print(" pip install setuptools")
    exit()

config = {
    "name": "myproject",
    "version": "0.0.0",
    "author": "Christiaan F Rademan",
    "author_email": "chris@fwiw.co.za",
    "description": "MyProject Package Example",
    "license": "BSD 3-Clause",
    "keywords": "another example",
    "url": "http://www.fwiw.co.za",
    "packages": find_packages(),
    "include_package_data": True,
    "classifiers": [
        "Topic :: Software Development :: Libraries :: Application Frameworks",
        "Environment :: Other Environment",
        "Intended Audience :: Information Technology",
        "Intended Audience :: System Administrators",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: BSD License",
        "Operating System :: POSIX :: Linux",
        "Programming Language :: Python",
        "Programming Language :: Python :: 2.7"
        ]
    }

# allow setup.py to be run from any path
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))

if os.path.exists(os.path.join(os.path.dirname(__file__),
                               'requirements.txt')):
    with open(os.path.join(os.path.dirname(__file__),
                           'requirements.txt')) as x:
        requirements = x.read().splitlines()
else:
    requirements = []

with open(os.path.join(os.path.dirname(__file__), 'README.rst')) as x:
    readme = x.read()

print("%s %s\n" % (config['name']))

setup(
    install_requires=requirements,
    long_description=readme,
    **config
)

You can use setup.py to register your project on PyPI which is explained later or use it to remove, install your package.

$ python setup.py install

will install the package

$ python setup.py develop

The develop will not install the package but it will create a .egg-link in the deployment directory back to the project source code directory.

So it's like installing but instead of copying to the site-packages it adds a symbolic link (the .egg-link acts as a multiplatform symbolic link).

That way you can edit the source code and see the changes directly without having to reinstall every time that you make a little change. This is useful when you are the developer of that project hence the name develop.

setup.cfg

This tells PyPI where your README file is.

[metadata]
description-file = README.rst

LICENSE.txt

This file will contain whichver license you want your code to have. I tend to use the BSD 3-Clause license.

What is PyPI?

From the official website:

PyPI — the Python Package Index

The Python Package Index is a repository of software for the Python programming language.

Upload your code on PyPI. It's a big list of python packages that you absolutely must submit your package to for it to be easily one-line installable.

PyPI Account

On PyPI Live and also on PyPI Test, you must create an account in order to be able to upload your code. I recommend using the same email/password for both accounts, just to make your life easier when it comes time to push.

~.pypirc configuration file

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=your_username

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username

Upload your package to PyPI Test

This will attempt to register your package against PyPI's test server, just to make sure you've set up everything correctly.

$ python setup.py register -r pypitest

Now upload your package

$ python setup.py sdist upload -r pypitest

Upload your package to PyPI Test

Simply run above commands again and replace pypitest with pypi

Error while Uploading

If you receive an error TypeError: cannot concatenate 'str' and 'NoneType' objects this is a known bug. Its because the upload is not prompting for the password.

To work around this register and upload at the same time like so:

$ python setup.py sdist register -r pypi upload -r pypi

VIM Cheat Sheet.

VI / VIM is a powerful editor and I built this small reference of commands that can be used.

VIM Typically functions in different modes.

Command mode

Possibly the most important thing to know is that when you're in command mode you can't insert text immediately. You first need to issue an insert, append, or open command to insert text.

Visual mode

Visual mode is used for copying and pasting for example.

Insert mode

Insert mode is used to add and append to the file.

Last line mode

You can only get to last line mode from command mode, and you get into last line mode by pressing the colon key.

Cheat Sheet / Commands

:help keyword - open help for keyword

:close - close current pane

:wq or :x or ZZ - write (save) and quit

:q - quit (fails if there are unsaved changes)

:q! or ZQ - quit and throw away unsaved changes

Working files

:e filename - edit another file

:o file - open file

:saveas file - save file as

:w - write (save) the file, but don't exit

:Explore - Directory listing

Cursor Movement

h - move cursor left

j - move cursor down

k - move cursor up

l - move cursor right

H - move to top of screen

M - move to middle of screen

L - move to bottom of screen

w - jump forwards to the start of a word

W - jump forwards to the start of a word (words can contain punctuation)

e - jump forwards to the end of a word

E - jump forwards to the end of a word (words can contain punctuation)

b - jump backwards to the start of a word

B - jump backwards to the start of a word (words can contain punctuation)

0 - jump to the start of the line

^ - jump to the first non

$ - jump to the end of the line

g_ - jump to the last non

gg - go to the first line of the document

G - go to the last line of the document

5G - go to line 5

fx - jump to next occurrence of character x

tx - jump to before next occurrence of character x

} - jump to next paragraph (or function/block, when editing code)

{ - jump to previous paragraph (or function/block, when editing code)

zz - center cursor on screen

Ctrl + b - move back one full screen

Ctrl + f - move forward one full screen

Ctrl + d - move forward 1/2 a screen

Ctrl + u - move back 1/2 a screen

Tip Prefix a cursor movement command with a number to repeat it. For example, 4j moves down 4 lines.

Insert mode

i - insert before the cursor

I - insert at the beginning of the line

a - insert (append) after the cursor

A - insert (append) at the end of the line

o - append (open) a new line below the current line

O - append (open) a new line above the current line

ea - insert (append) at the end of the word

Esc - exit insert mode

Editing

r - replace a single character

J - join line below to the current one

cc - change (replace) entire line

cw - change (replace) to the end of the word

c$ - change (replace) to the end of the line

s - delete character and substitute text

S - delete line and substitute text (same as cc)

xp - transpose two letters (delete and paste)

u - undo

Ctrl + r - redo

Marking Text / Visual Mode

v - start visual mode, mark lines, then do a command (like y

V - start linewise visual mode

o - move to other end of marked area

Ctrl + v - start visual block mode

O - move to other corner of block

aw - mark a word

ab - a block with ()

aB - a block with {}

ib - inner block with ()

iB - inner block with {}

Esc - exit visual mode

Visual Commands

> - shift text right

< - shift text left

y - yank (copy) marked text

d - delete marked text

~ - switch case

Cut and Paste

yy - yank (copy) a line

2yy - yank (copy) 2 lines

yw - yank (copy) the characters of the word from the cursor position to the start of the next word

y$ - yank (copy) to end of line

p - put (paste) the clipboard after cursor

P - put (paste) before cursor

dd - delete (cut) a line

2dd - delete (cut) 2 lines

dw - delete (cut) the characters of the word from the cursor position to the start of the next word

D - delete (cut) to the end of the line

d$ - delete (cut) to the end of the line

x - delete (cut) character

Search and replace

/pattern - search for pattern

?pattern - search backward for pattern

vpattern - 'very magic' pattern: non

n - repeat search in same direction

N - repeat search in opposite direction

:%s/old/new/g - replace all old with new throughout file

:%s/old/new/gc - replace all old with new throughout file with confirmations

:noh - remove highlighting of search matches

Multiple Windows

:split filename - split window and load another file

ctrl-w up arrow - move to window above

ctrl-w left arrow - move to window left

ctrl-w up right arrow - move to window right

ctrl-w up down arrow - move to window below

ctrl-w ctrl-w - move cursor to another window (cycle)

ctrl-w_ - maximize current window

ctrl-w= - make all equal size

10 ctrl-w+ - increase window size by 10 lines

:vsplit file - vertical split

:sview file - same as split, but readonly

:hide - close current window

:only - keep only this window open

:ls - show current buffers

:b 2 - open buffer #2 in this window

GIT 101

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

Based on this article I am using GITHUB http://www.github.com for hosting my repositories securely online. Code is hosted on GITHUB makes the code easy to browse, download, fork, etc.

Clone a project

Cloning a project is creating a local copy on your host of a remote repository.

$ git clone https://github.com/vision1983/nfw.git

Joining / Fork a project

On GITHUB you can create a fork which is a copy of repository hosted on GITHUB within your own GITHUB space.

Once you pushed your changes to own copy on GITHUB you can do a pull request on GITHUB which will inform the maintainers of the original project that you wish to merge your changes upstream.

Creating your own GITHUB project

Create a repository on GITHUB without initializing.

  • Enter your source code directory
  • Create a new repository with empty README.md or README.rst.
$ git init
$ git add README.md
$ git commit -m "first commit"
$ git remote add origin https://github.com/vision1983/test.git
$ git push -u origin master

or push exisiting repository to GITHUB

git remote add origin https://github.com/vision1983/test.git
git push -u origin master

Working with local repository

To commit changes you need to stage the changes using the following commands:

Add file to repository

$ git add file

Remove file from repository

$ git del file

Before committing you can undo or checkout original file

$ git checkout file

Commit changes locally

$ git commit -a

Finally upload or push to GITHUB

$ git push

Syncing a fork

Sync a fork of a repository to keep it up-to-date with the upstream repository.

Before you can sync your fork with an upstream repository, you must configure a remote that points to the upstream repository in GITHUB.

List current configured remote repository for your fork

$ git remote -v
origin  https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
origin  https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)

Specify a new remote upstream repository that will be synced with the fork

$ git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.

Verify the new upstream repository you've specified for your fork

$ git remote -v
origin https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
origin https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)
upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (fetch)
upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (push)

Fetch the branches and their respective commits from the upstream repository. Commits to master will be stored in a local branch, upstream/master

$ git fetch upstream
remote: Counting objects: 75, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 62 (delta 27), reused 44 (delta 9)
Unpacking objects: 100% (62/62), done.
From https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY
* [new branch]      master     -> upstream/master

Check out your fork's local master branch

$ git checkout master
Switched to branch 'master'

Merge the changes from upstream/master into your local master branch. This brings your fork's master branch into sync with the upstream repository, without losing your local changes

$ git merge upstream/master

GIT Tags

Git has the ability to tag specific points in history as being important. Typically people use this functionality to mark release points (1.0.0, and so on)

List your tags

$ git tag
0.0.1
0.0.2

Git uses two main types of tags

  • lightweight
  • and annotated.

Annotated Tags

Annotated as per GIT Documentation

Annotated tags, however, are stored as full objects in the Git database. They’re checksummed; contain the tagger name, email, and date; have a tagging message; and can be signed and verified with GNU Privacy Guard (GPG). It’s generally recommended that you create annotated tags so you can have all this information; but if you want a temporary tag or for some reason don’t want to keep the other information, lightweight tags are available too.

Creating an annotated tag

The easiest way is to specify -a when you run the tag command.

$ git tag -a 0.0.2 -m "my version 0.0.2"
$ git tag
0.0.0
0.0.1
0.0.2

The -m specifies a tagging message, which is stored with the tag.

You can see the tag data along with the commit that was tagged by using the git show command

$ git show 0.0.2

Lightweight Tags

Lightweight as per GIT Documentation

A lightweight tag is very much like a branch that doesn’t change. It just points to a specific commit.

To create a lightweight tag, don’t supply the -a, -s, or -m option

$ git tag 0.0.3
$ git tag
0.0.0
0.0.1
0.0.2
0.0.3

When you run git show on the tag, you don’t see the extra tag information as per with annotated tag.

Tagging Later

You can also tag commits after you’ve moved past them.

Retrieve commit history looks like this

$ git log --pretty=oneline

Tag based on checksum from history

$ git tag -a 0.0.6 [checksum]

Sharing Tags

By default, the git push command doesn’t transfer tags to remote servers. You will have to explicitly push tags to a shared server after you have created them.

$ git push origin [tagname]

If you have a lot of tags that you want to push up at once, you can

$ git push origin --tags

Delete Tags

Delete tag locally

$ git tag -d 0.0.1

Delete tag from upstream

$ git push origin :refs/tags/1.0.5

Now, when someone else clones or pulls from your repository, they will get all your tags as well.

Managing Branches

In your GITHUB project, you need to keep your master branch clean, by clean I mean without any changes, like that you can create at any time a branch from your master. Each time, that you want to commit a bug or a feature, you need to create a branch for it, which will be a copy of your master branch.

When you do a pull request on a branch, you can continue to work on another branch and make another pull request on this other branch.

Before creating a new branch, pull the changes from upstream. Your master needs to be up to date.

Create the branch on your local machine and switch in this branch

$ git checkout -b [name_of_your_new_branch]

Push the branch on github

$ git push origin [name_of_your_new_branch]

When you want to commit something in your branch, be sure to be in your branch.

You can see all branches created by using

$ git branch

See details on branches for origin/GITHUB

$ git remote show origin

Switch to a branch

$ git checkout [name_of_your_new_branch]

Pull updates from branch on GITHUB

$ git pull origin [name_of_your_new_branch]

Push updates to your branch on GITHUB

$ git push origin [name_of_your_new_branch]

Delete a branch on your local filesystem

$ git branch -d [name_of_your_new_branch]

Force the deletion of local branch on your filesystem

$ git branch -D [name_of_your_new_branch]

Delete the branch on github

$ git push origin :[name_of_your_new_branch]