{
 "metadata": {
  "name": "numpy"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# A brief introduction to Numpy\n",
      "[Numpy](http://numpy.scipy.org/) is the fundamental library for scientific computing in Python. It contains list like objects that work like arrays, matrices, and data tables. This is how scientists typically expect data to behave. Numpy also provides linear algebra, Fourier transforms, random number generation, and tools for integrating C/C++ and Fortran code.\n",
      "\n",
      "If you primarily want to work with tables of data, [Pandas](pandas.ipynb), which depends on Numpy, is probably the module that you want to start with."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Numpy Array Basics\n",
      "#### Creating a Numpy array"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "\n",
      "example_array = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
      "example_array"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 1,
       "text": [
        "array([[1, 2, 3],\n",
        "       [4, 5, 6],\n",
        "       [7, 8, 9]])"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Indexing an array"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "example_array[1, 1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 2,
       "text": [
        "5"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Slicing an array"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "example_array[:, 0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "array([1, 4, 7])"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "example_array[1, :]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "array([4, 5, 6])"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "example_array[1:3, 1:3]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "array([[5, 6],\n",
        "       [8, 9]])"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Subsetting an array"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "array1 = np.array([1, 1, 1, 2, 2, 2, 1])\n",
      "array2 = np.array([1, 2, 3, 4, 5, 6, 7])\n",
      "array2[array1==1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "array([1, 2, 3, 7])"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "array3 = np.array(['a', 'a', 'a', 'b', 'b', 'b', 'b'])\n",
      "array2[(array1==1) & (array3=='a')]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "array([1, 2, 3])"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Math\n",
      "### Arrays\n",
      "Math on arrays is vectorized and behaves exactly like most scientists would expect"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "array1 = np.array([1, 1, 1, 2, 2, 2, 1])\n",
      "array2 = np.array([1, 2, 3, 4, 5, 6, 7])\n",
      "\n",
      "array1 * 2 + 1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "array([3, 3, 3, 5, 5, 5, 3])"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "array1 * array2"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "array([ 1,  2,  3,  8, 10, 12,  7])"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Linear algebra (matrices)\n",
      "Linear algebra is done using a different data structure called a matrix."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "matrix1 = np.matrix([[1, 2, 3], [4, 5, 6]])\n",
      "matrix2 = np.matrix([1, 2, 3])\n",
      "matrix1 * matrix2.transpose()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "matrix([[14],\n",
        "        [32]])"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Importing and Exporting Data\n",
      "The numpy function genfromtxt is a powerful way to import text data.\n",
      "It can use different delimiters, skip header rows, control the type of imported data, give columns of data names, and a number of other useful goodies. See the [documentation](http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html) for a full list of features of run help(np.genfromtxt) from the Python shell (after importing the module of course)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Basic Import and Export\n",
      "#### Import\n",
      "Basic imports using Numpy will treat all data as floats.\n",
      "If we're doing a basic import we'll typically want to skip the header row (since it's generally not composed of numbers."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = np.genfromtxt('./data/examp_data.txt', delimiter=',', skip_header=1)\n",
      "data"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "array([[ 1. ,  2. ,  3. ],\n",
        "       [ 2. ,  2.4,  6. ],\n",
        "       [ 3. ,  1.9,  8. ]])"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Export"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "np.savetxt('./data/examp_output.txt', data, delimiter=',')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Importing Data Tables\n",
      "Lots of scientific data comes in the form of tables, with one row per observation, and one column per thing observed.\n",
      "Often the different columns to have different types (including text).\n",
      "The best way to work with this type of data is in a Structured Array."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Import\n",
      "To do this we let Numpy automatically detect the data types in each column using the optional argument ``dtype=None``.\n",
      "We can also use an existing header row as the names for the columns using the optional arugment ``Names=True``."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = np.genfromtxt('./data/examp_data_species_mass.txt', dtype=None, names=True, delimiter=',')\n",
      "data"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "array([(1, 'DS', 125), (1, 'DM', 70), (2, 'DM', 55), (1, 'CB', 40),\n",
        "       (2, 'DS', 110), (1, 'CB', 45)], \n",
        "      dtype=[('site', '<i8'), ('species', '|S2'), ('mass', '<i8')])"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Export\n",
      "The easiest way to export a structured array is to treat it like a list of lists and export it using the csv module using a function like this."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "def export_to_csv(data, filename):\n",
      "    outputfile = open(filename, 'wb')\n",
      "    datawriter = csv.writer(outputfile)\n",
      "    datawriter.writerows(data)\n",
      "    outputfile.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Structured Arrays\n",
      "-----------------\n",
      "If we import data into a Structured Array we can do a lot of things that we often want to do with scientific data."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Selecting columns by name"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = np.genfromtxt('./data/examp_data_species_mass.txt', dtype=None, names=True, delimiter=',')\n",
      "print data\n",
      "data['species']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[(1, 'DS', 125) (1, 'DM', 70) (2, 'DM', 55) (1, 'CB', 40) (2, 'DS', 110)\n",
        " (1, 'CB', 45)]"
       ]
      },
      {
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "array(['DS', 'DM', 'DM', 'CB', 'DS', 'CB'], \n",
        "      dtype='|S2')"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Subset columns based on the values in other columns"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data['mass'][data['species'] == 'DM']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 16,
       "text": [
        "array([70, 55])"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data['mass'][(data['species'] == 'DM') & (data['site'] == 1)]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "array([70])"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Random number generation\n",
      "#### Random uniform (0 to 1)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "np.random.rand(3, 5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 18,
       "text": [
        "array([[ 0.03414585,  0.83900235,  0.93206285,  0.06820967,  0.70145045],\n",
        "       [ 0.552352  ,  0.76730225,  0.06316622,  0.71285231,  0.81976971],\n",
        "       [ 0.39709379,  0.71772434,  0.21598482,  0.96412023,  0.69841293]])"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Random normal (mean=0, stdev=1)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "np.random.randn(4, 2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "array([[ 0.04802043, -0.89025722],\n",
        "       [ 0.46246887,  1.11994326],\n",
        "       [-0.95655129,  0.76707094],\n",
        "       [-1.61019706, -0.21933367]])"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Random integers"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "min = 10\n",
      "max = 20\n",
      "np.random.randint(min, max, [10, 2])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "array([[13, 10],\n",
        "       [18, 10],\n",
        "       [10, 12],\n",
        "       [10, 16],\n",
        "       [17, 12],\n",
        "       [17, 13],\n",
        "       [19, 16],\n",
        "       [11, 17],\n",
        "       [11, 16],\n",
        "       [16, 18]])"
       ]
      }
     ],
     "prompt_number": 20
    }
   ],
   "metadata": {}
  }
 ]
}