A Python Book: Beginning Python, Advanced Python, and Python

A Python Book A Python Book: Beginning Python, Advanced Python, and Python Exercises Author: Dave Kuhlman Contact: dkuhlman@davekuhlman.org Address: http://www.davekuhlman.org Page 1 A Python Book Revision 1.3a Date December 15, 2013 Copyright Copyright (c) 2009 Dave Kuhlman. All Rights Reserved. This document is subject to the provisions of the Open Source MIT License http://www.opensource.org/licenses/mit­license.php. Abstract This document is a self­learning document for a course in Python programming. This course contains (1) a part for beginners, (2) a discussion of several advanced topics that are of interest to Python programmers, and (3) a Python workbook with lots of exercises. Page 2 A Python Book Contents 1 Part 1 ­­ Beginning Python...........................................................................................10 1.1 Introductions Etc...................................................................................................10 1.1.1 Resources.......................................................................................................11 1.1.2 A general description of Python....................................................................12 1.1.3 Interactive Python..........................................................................................15 1.2 Lexical matters......................................................................................................15 1.2.1 Lines..............................................................................................................15 1.2.2 Comments......................................................................................................16 1.2.3 Names and tokens..........................................................................................16 1.2.4 Blocks and indentation..................................................................................16 1.2.5 Doc strings.....................................................................................................17 1.2.6 Program structure..........................................................................................17 1.2.7 Operators.......................................................................................................18 1.2.8 Also see.........................................................................................................19 1.2.9 Code evaluation.............................................................................................19 1.3 Statements and inspection ­­ preliminaries...........................................................20 1.4 Built­in data­types.................................................................................................21 1.4.1 Numeric types................................................................................................21 1.4.2 Tuples and lists..............................................................................................21 1.4.3 Strings............................................................................................................24 1.4.3.1 The new string.format method...............................................................26 1.4.3.2 Unicode strings......................................................................................27 1.4.4 Dictionaries....................................................................................................29 1.4.5 Files...............................................................................................................32 1.4.6 Other built­in types........................................................................................35 1.4.6.1 The None value/type..............................................................................35 1.4.6.2 Boolean values.......................................................................................36 1.4.6.3 Sets and frozensets.................................................................................36 1.5 Functions and Classes ­­ A Preview......................................................................36 1.6 Statements.............................................................................................................37 1.6.1 Assignment statement....................................................................................37 1.6.2 import statement............................................................................................39 1.6.3 print statement...............................................................................................41 1.6.4 if: elif: else: statement...................................................................................43 1.6.5 for: statement.................................................................................................44 1.6.6 while: statement.............................................................................................48 Page 3 A Python Book 1.6.7 continue and break statements.......................................................................48 1.6.8 try: except: statement.....................................................................................49 1.6.9 raise statement...............................................................................................51 1.6.10 with: statement.............................................................................................52 1.6.10.1 Writing a context manager...................................................................52 1.6.10.2 Using the with: statement....................................................................53 1.6.11 del................................................................................................................54 1.6.12 case statement..............................................................................................55 1.7 Functions, Modules, Packages, and Debugging....................................................55 1.7.1 Functions.......................................................................................................55 1.7.1.1 The def statement...................................................................................55 1.7.1.2 Returning values....................................................................................55 1.7.1.3 Parameters..............................................................................................56 1.7.1.4 Arguments..............................................................................................56 1.7.1.5 Local variables.......................................................................................57 1.7.1.6 Other things to know about functions....................................................57 1.7.1.7 Global variables and the global statement.............................................58 1.7.1.8 Doc strings for functions.......................................................................60 1.7.1.9 Decorators for functions........................................................................60 1.7.2 lambda...........................................................................................................61 1.7.3 Iterators and generators.................................................................................62 1.7.4 Modules.........................................................................................................67 1.7.4.1 Doc strings for modules.........................................................................68 1.7.5 Packages........................................................................................................68 1.8 Classes...................................................................................................................69 1.8.1 A simple class................................................................................................69 1.8.2 Defining methods..........................................................................................70 1.8.3 The constructor..............................................................................................70 1.8.4 Member variables..........................................................................................70 1.8.5 Calling methods.............................................................................................71 1.8.6 Adding inheritance........................................................................................71 1.8.7 Class variables...............................................................................................72 1.8.8 Class methods and static methods.................................................................72 1.8.9 Properties.......................................................................................................74 1.8.10 Interfaces.....................................................................................................75 1.8.11 New­style classes.........................................................................................75 1.8.12 Doc strings for classes.................................................................................77 1.8.13 Private members..........................................................................................77 1.9 Special Tasks.........................................................................................................77 1.9.1 Debugging tools.............................................................................................77 Page 4 A Python Book 1.9.2 File input and output......................................................................................78 1.9.3 Unit tests........................................................................................................80 1.9.3.1 A simple example..................................................................................80 1.9.3.2 Unit test suites........................................................................................81 1.9.3.3 Additional unittest features....................................................................83 1.9.3.4 Guidance on Unit Testing......................................................................85 1.9.4 doctest............................................................................................................85 1.9.5 The Python database API..............................................................................87 1.9.6 Installing Python packages............................................................................88 1.10 More Python Features and Exercises..................................................................89 2 Part 2 ­­ Advanced Python............................................................................................90 2.1 Introduction ­­ Python 201 ­­ (Slightly) Advanced Python Topics.......................90 2.2 Regular Expressions..............................................................................................90 2.2.1 Defining regular expressions.........................................................................90 2.2.2 Compiling regular expressions......................................................................91 2.2.3 Using regular expressions..............................................................................91 2.2.4 Using match objects to extract a value..........................................................92 2.2.5 Extracting multiple items..............................................................................93 2.2.6 Replacing multiple items...............................................................................94 2.3 Iterator Objects......................................................................................................96 2.3.1 Example ­ A generator function....................................................................98 2.3.2 Example ­ A class containing a generator method......................................100 2.3.3 Example ­ An iterator class.........................................................................102 2.3.4 Example ­ An iterator class that uses yield.................................................104 2.3.5 Example ­ A list comprehension.................................................................105 2.3.6 Example ­ A generator expression..............................................................105 2.4 Unit Tests............................................................................................................106 2.4.1 Defining unit tests........................................................................................106 2.4.1.1 Create a test class.................................................................................106 2.5 Extending and embedding Python......................................................................109 2.5.1 Introduction and concepts............................................................................109 2.5.2 Extension modules.......................................................................................110 2.5.3 SWIG...........................................................................................................112 2.5.4 Pyrex............................................................................................................115 2.5.5 SWIG vs. Pyrex...........................................................................................120 2.5.6 Cython.........................................................................................................120 2.5.7 Extension types............................................................................................122 2.5.8 Extension classes.........................................................................................122 2.6 Parsing.................................................................................................................122 2.6.1 Special purpose parsers...............................................................................123 Page 5 A Python Book 2.6.2 Writing a recursive descent parser by hand.................................................124 2.6.3 Creating a lexer/tokenizer with Plex...........................................................131 2.6.4 A survey of existing tools............................................................................141 2.6.5 Creating a parser with PLY.........................................................................141 2.6.6 Creating a parser with pyparsing.................................................................148 2.6.6.1 Parsing comma­delimited lines............................................................148 2.6.6.2 Parsing functors...................................................................................149 2.6.6.3 Parsing names, phone numbers, etc.....................................................150 2.6.6.4 A more complex example....................................................................151 2.7 GUI Applications................................................................................................153 2.7.1 Introduction.................................................................................................153 2.7.2 PyGtk...........................................................................................................153 2.7.2.1 A simple message dialog box..............................................................153 2.7.2.2 A simple text input dialog box.............................................................156 2.7.2.3 A file selection dialog box...................................................................158 2.7.3 EasyGUI......................................................................................................160 2.7.3.1 A simple EasyGUI example................................................................161 2.7.3.2 An EasyGUI file open dialog example................................................161 2.8 Guidance on Packages and Modules...................................................................161 2.8.1 Introduction.................................................................................................161 2.8.2 Implementing Packages...............................................................................162 2.8.3 Using Packages............................................................................................162 2.8.4 Distributing and Installing Packages...........................................................162 2.9 End Matter...........................................................................................................164 2.9.1 Acknowledgements and Thanks..................................................................164 2.9.2 See Also.......................................................................................................164 3 Part 3 ­­ Python Workbook.........................................................................................165 3.1 Introduction.........................................................................................................165 3.2 Lexical Structures................................................................................................165 3.2.1 Variables and names....................................................................................165 3.2.2 Line structure...............................................................................................167 3.2.3 Indentation and program structure...............................................................168 3.3 Execution Model.................................................................................................169 3.4 Built­in Data Types.............................................................................................170 3.4.1 Numbers......................................................................................................170 3.4.1.1 Literal representations of numbers......................................................171 3.4.1.2 Operators for numbers.........................................................................173 3.4.1.3 Methods on numbers............................................................................175 3.4.2 Lists.............................................................................................................175 3.4.2.1 Literal representation of lists...............................................................176 Page 6 A Python Book 3.4.2.2 Operators on lists.................................................................................178 3.4.2.3 Methods on lists...................................................................................178 3.4.2.4 List comprehensions............................................................................180 3.4.3 Strings..........................................................................................................182 3.4.3.1 Characters............................................................................................183 3.4.3.2 Operators on strings.............................................................................184 3.4.3.3 Methods on strings...............................................................................185 3.4.3.4 Raw strings..........................................................................................187 3.4.3.5 Unicode strings....................................................................................188 3.4.4 Dictionaries..................................................................................................190 3.4.4.1 Literal representation of dictionaries...................................................190 3.4.4.2 Operators on dictionaries.....................................................................191 3.4.4.3 Methods on dictionaries.......................................................................192 3.4.5 Files.............................................................................................................195 3.4.6 A few miscellaneous data types..................................................................197 3.4.6.1 None.....................................................................................................197 3.4.6.2 The booleans True and False...............................................................197 3.5 Statements...........................................................................................................198 3.5.1 Assignment statement..................................................................................198 3.5.2 print statement.............................................................................................200 3.5.3 if: statement exercises..................................................................................201 3.5.4 for: statement exercises...............................................................................202 3.5.5 while: statement exercises...........................................................................205 3.5.6 break and continue statements.....................................................................206 3.5.7 Exceptions and the try:except: and raise statements...................................207 3.6 Functions.............................................................................................................210 3.6.1 Optional arguments and default values.......................................................211 3.6.2 Passing functions as arguments...................................................................213 3.6.3 Extra args and keyword args.......................................................................214 3.6.3.1 Order of arguments (positional, extra, and keyword args)..................216 3.6.4 Functions and duck­typing and polymorphism...........................................216 3.6.5 Recursive functions.....................................................................................217 3.6.6 Generators and iterators...............................................................................219 3.7 Object­oriented programming and classes..........................................................223 3.7.1 The constructor............................................................................................224 3.7.2 Inheritance ­­ Implementing a subclass.......................................................225 3.7.3 Classes and polymorphism..........................................................................227 3.7.4 Recursive calls to methods..........................................................................228 3.7.5 Class variables, class methods, and static methods.....................................230 3.7.5.1 Decorators for classmethod and staticmethod.....................................233 Page 7 A Python Book 3.8 Additional and Advanced Topics........................................................................234 3.8.1 Decorators and how to implement them......................................................234 3.8.1.1 Decorators with arguments..................................................................235 3.8.1.2 Stacked decorators...............................................................................236 3.8.1.3 More help with decorators...................................................................238 3.8.2 Iterables.......................................................................................................239 3.8.2.1 A few preliminaries on Iterables..........................................................239 3.8.2.2 More help with iterables......................................................................240 3.9 Applications and Recipes....................................................................................240 3.9.1 XML ­­ SAX, minidom, ElementTree, Lxml..............................................241 3.9.2 Relational database access...........................................................................249 3.9.3 CSV ­­ comma separated value files...........................................................255 3.9.4 YAML and PyYAML..................................................................................256 3.9.5 Json..............................................................................................................258 4 Part 4 ­­ Generating Python Bindings for XML.........................................................260 4.1 Introduction.........................................................................................................260 4.2 Generating the code.............................................................................................261 4.3 Using the generated code to parse and export an XML document.....................263 4.4 Some command line options you might want to know.......................................263 4.5 The graphical front­end.......................................................................................264 4.6 Adding application­specific behavior.................................................................265 4.6.1 Implementing custom subclasses................................................................265 4.6.2 Using the generated "API" from your application......................................266 4.6.3 A combined approach..................................................................................267 4.7 Special situations and uses..................................................................................269 4.7.1 Generic, type­independent processing.........................................................269 4.7.1.1 Step 1 ­­ generate the bindings............................................................270 4.7.1.2 Step 2 ­­ add application­specific code................................................270 4.7.1.3 Step 3 ­­ write a test/driver harness.....................................................274 4.7.1.4 Step 4 ­­ run the test application..........................................................276 4.8 Some hints...........................................................................................................276 4.8.1 Children defined with maxOccurs greater than 1........................................276 4.8.2 Children defined with simple numeric types...............................................277 4.8.3 The type of an element's character content..................................................277 4.8.4 Constructors and their default values..........................................................277 Page 8 A Python Book Preface This book is a collection of materials that I've used when conducting Python training and also materials from my Web site that are intended for self­instruction. You may prefer a machine readable copy of this book. You can find it in various formats here: ? HTML – http://www.davekuhlman.org/python_book_01.html ? PDF ­­ http://www.davekuhlman.org /python_book_01.pdf ? ODF/OpenOffice ­­ http://www.davekuhlman.org /python_book_01.odt And, let me thank the students in my Python classes. Their questions and suggestions were a great help in the preparation of these materials. Page 9 A Python Book 1 Part 1 ­­ Beginning Python 1.1 Introductions Etc Introductions Practical matters: restrooms, breakroom, lunch and break times, etc. Starting the Python interactive interpreter. Also, IPython and Idle. Running scripts Editors ­­ Choose an editor which you can configure so that it indents with 4 spaces, not tab characters. For a list of editors for Python, see: http://wiki.python.org/moin/PythonEditors. A few possible editors: ? SciTE ­­ http://www.scintilla.org/SciTE.html. ? MS Windows only ­­ (1) TextPad ­­ http://www.textpad.com; (2) UltraEdit ­­ http://www.ultraedit.com/. ? Jed ­­ See http://www.jedsoft.org/jed/. ? Emacs ­­ See http://www.gnu.org/software/emacs/ and http://www.xemacs.org/faq/xemacs­faq.html. ? jEdit ­­ Requires a bit of customization for Python ­­ See http://jedit.org. ? Vim ­­ http://www.vim.org/ ? Geany ­­ http://www.geany.org/ ? And many more. Interactive interpreters: ? python ? ipython ? Idle IDEs ­­ Also see http://en.wikipedia.org/wiki/List_of_integrated_development_environments_for_Python: ? PyWin ­­ MS Windows only. Available at: http://sourceforge.net/projects/pywin32/. ? WingIDE ­­ See http://wingware.com/wingide/. ? Eclipse ­­ http://eclipse.org/. There is a plug­in that supports Python. ? Kdevelop ­­ Linux/KDE ­­ See http://www.kdevelop.org/. ? Eric ­­ Linux KDE? ­­ See http://eric­ide.python­projects.org/index.html ? Emacs and SciTE will evaluate a Python buffer within the editor. Page 10