{ "metadata": { "name": "TideDataDemo" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "#NumPy Demo\n", "\n", "We'll be using [this CSV file](https://raw.github.com/gist/4049563/385dd8c10e8e74eb05d8d83266a2acc8e2ccd7f7/BatteryParkTideData.csv) of tide level data for this demo. The data are from Battery Park, New York City on October 29-30, 2012, during Hurricane Sandy. The data in the CSV are a reformatted and cleaned version of [this data from NOAA](http://tidesandcurrents.noaa.gov/data_menu.shtml?plot_backup=1&bdate=20121029&edate=20121030&datum=6&unit=1&shift=d&stn=8518750+The+Battery%2C+NY&type=Tide+Data&format=View+Data)." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import numpy as np" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading Data\n", "\n", "Basic read with [numpy.genfromtxt](http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt). Returns a 2d array of floats." ] }, { "cell_type": "code", "collapsed": false, "input": [ "!head -n 5 BatteryParkTideData.csv" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "TimeOffsetHours,Pred6,Backup,Acoustc\r\n", "0.0,1.5900000000000001,4.6799999999999997,4.6500000000000004\r\n", "0.10000000000000001,1.5,4.5499999999999998,4.54\r\n", "0.20000000000000001,1.3999999999999999,4.46,4.4400000000000004\r\n", "0.29999999999999999,1.3100000000000001,4.3600000000000003,4.3300000000000001\r\n" ] } ], "prompt_number": 82 }, { "cell_type": "code", "collapsed": false, "input": [ "data = np.genfromtxt('BatteryParkTideData.csv', delimiter=',', skip_header=1, missing='NA')" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 2 }, { "cell_type": "code", "collapsed": false, "input": [ "data" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 3, "text": [ "array([[ 0. , 1.59, 4.68, 4.65],\n", " [ 0.1 , 1.5 , 4.55, 4.54],\n", " [ 0.2 , 1.4 , 4.46, 4.44],\n", " ..., \n", " [ 47.7 , 3.25, 4.32, 4.5 ],\n", " [ 47.8 , 3.14, 4.22, 4.39],\n", " [ 47.9 , 3.03, 4.12, 4.28]])" ] } ], "prompt_number": 3 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Array Properties" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print 'Shape: ', data.shape\n", "print 'Size: ', data.size\n", "print 'Number of dimensions: ', data.ndim\n", "print 'Data type: ', data.dtype" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Shape: (480, 4)\n", "Size: 1920\n", "Number of dimensions: 2\n", "Data type: float64\n" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "data[0]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 5, "text": [ "array([ 0. , 1.59, 4.68, 4.65])" ] } ], "prompt_number": 5 }, { "cell_type": "code", "collapsed": false, "input": [ "data[0, 1]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 6, "text": [ "1.5900000000000001" ] } ], "prompt_number": 6 }, { "cell_type": "code", "collapsed": false, "input": [ "data[:, 1]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 7, "text": [ "array([ 1.59, 1.5 , 1.4 , 1.31, 1.22, 1.13, 1.04, 0.95, 0.87,\n", " 0.78, 0.7 , 0.62, 0.55, 0.48, 0.41, 0.34, 0.28, 0.23,\n", " 0.18, 0.14, 0.1 , 0.08, 0.06, 0.05, 0.05, 0.06, 0.08,\n", " 0.11, 0.14, 0.19, 0.25, 0.31, 0.39, 0.47, 0.56, 0.66,\n", " 0.76, 0.87, 0.98, 1.1 , 1.22, 1.35, 1.48, 1.61, 1.74,\n", " 1.88, 2.01, 2.15, 2.28, 2.41, 2.55, 2.68, 2.81, 2.94,\n", " 3.06, 3.19, 3.31, 3.43, 3.55, 3.66, 3.78, 3.89, 3.99,\n", " 4.1 , 4.2 , 4.3 , 4.39, 4.48, 4.57, 4.66, 4.73, 4.81,\n", " 4.89, 4.95, 5.02, 5.08, 5.13, 5.18, 5.23, 5.27, 5.3 ,\n", " 5.33, 5.35, 5.37, 5.38, 5.38, 5.38, 5.37, 5.36, 5.34,\n", " 5.31, 5.28, 5.24, 5.19, 5.13, 5.07, 5. , 4.93, 4.85,\n", " 4.76, 4.67, 4.57, 4.47, 4.37, 4.26, 4.15, 4.03, 3.92,\n", " 3.8 , 3.68, 3.56, 3.44, 3.32, 3.21, 3.09, 2.97, 2.86,\n", " 2.74, 2.63, 2.52, 2.41, 2.3 , 2.2 , 2.09, 1.98, 1.88,\n", " 1.78, 1.68, 1.57, 1.47, 1.37, 1.27, 1.17, 1.08, 0.98,\n", " 0.89, 0.8 , 0.71, 0.62, 0.54, 0.46, 0.39, 0.32, 0.26,\n", " 0.21, 0.16, 0.12, 0.09, 0.07, 0.06, 0.05, 0.06, 0.08,\n", " 0.1 , 0.13, 0.18, 0.23, 0.29, 0.36, 0.44, 0.52, 0.61,\n", " 0.71, 0.81, 0.91, 1.02, 1.13, 1.25, 1.36, 1.49, 1.6 ,\n", " 1.73, 1.85, 1.97, 2.09, 2.21, 2.33, 2.45, 2.57, 2.68,\n", " 2.8 , 2.91, 3.02, 3.13, 3.23, 3.33, 3.44, 3.53, 3.63,\n", " 3.72, 3.81, 3.9 , 3.98, 4.06, 4.14, 4.21, 4.28, 4.34,\n", " 4.41, 4.46, 4.51, 4.56, 4.6 , 4.64, 4.67, 4.69, 4.71,\n", " 4.73, 4.74, 4.74, 4.74, 4.72, 4.71, 4.69, 4.65, 4.62,\n", " 4.57, 4.52, 4.46, 4.4 , 4.33, 4.25, 4.17, 4.08, 3.99,\n", " 3.89, 3.79, 3.68, 3.57, 3.46, 3.35, 3.23, 3.11, 3. ,\n", " 2.88, 2.76, 2.65, 2.53, 2.42, 2.31, 2.2 , 2.1 , 1.99,\n", " 1.89, 1.79, 1.7 , 1.6 , 1.51, 1.42, 1.33, 1.25, 1.16,\n", " 1.08, 1. , 0.92, 0.85, 0.77, 0.7 , 0.63, 0.56, 0.5 ,\n", " 0.44, 0.38, 0.33, 0.29, 0.24, 0.21, 0.18, 0.16, 0.15,\n", " 0.14, 0.14, 0.16, 0.18, 0.21, 0.25, 0.3 , 0.36, 0.43,\n", " 0.5 , 0.59, 0.68, 0.78, 0.89, 1. , 1.12, 1.24, 1.37,\n", " 1.5 , 1.63, 1.77, 1.91, 2.05, 2.19, 2.32, 2.46, 2.6 ,\n", " 2.73, 2.87, 3. , 3.13, 3.25, 3.38, 3.5 , 3.62, 3.73,\n", " 3.84, 3.95, 4.06, 4.16, 4.25, 4.35, 4.44, 4.52, 4.6 ,\n", " 4.68, 4.76, 4.83, 4.89, 4.95, 5.01, 5.07, 5.11, 5.16,\n", " 5.2 , 5.23, 5.26, 5.29, 5.31, 5.32, 5.33, 5.33, 5.33,\n", " 5.32, 5.31, 5.29, 5.26, 5.23, 5.19, 5.14, 5.1 , 5.04,\n", " 4.97, 4.9 , 4.83, 4.75, 4.66, 4.57, 4.47, 4.37, 4.27,\n", " 4.16, 4.05, 3.93, 3.81, 3.69, 3.57, 3.45, 3.33, 3.21,\n", " 3.09, 2.97, 2.86, 2.74, 2.63, 2.51, 2.4 , 2.29, 2.19,\n", " 2.08, 1.98, 1.88, 1.78, 1.68, 1.58, 1.49, 1.39, 1.3 ,\n", " 1.21, 1.12, 1.03, 0.94, 0.85, 0.77, 0.69, 0.6 , 0.53,\n", " 0.46, 0.39, 0.33, 0.27, 0.22, 0.18, 0.14, 0.11, 0.09,\n", " 0.08, 0.08, 0.09, 0.1 , 0.13, 0.16, 0.2 , 0.26, 0.32,\n", " 0.39, 0.47, 0.55, 0.64, 0.74, 0.84, 0.95, 1.06, 1.17,\n", " 1.29, 1.41, 1.53, 1.66, 1.78, 1.9 , 2.02, 2.15, 2.27,\n", " 2.39, 2.5 , 2.62, 2.73, 2.84, 2.95, 3.05, 3.16, 3.26,\n", " 3.36, 3.45, 3.54, 3.63, 3.71, 3.8 , 3.87, 3.95, 4.02,\n", " 4.09, 4.15, 4.22, 4.27, 4.32, 4.37, 4.42, 4.46, 4.49,\n", " 4.52, 4.55, 4.57, 4.58, 4.6 , 4.6 , 4.6 , 4.59, 4.58,\n", " 4.56, 4.54, 4.51, 4.48, 4.43, 4.38, 4.33, 4.27, 4.2 ,\n", " 4.13, 4.05, 3.96, 3.87, 3.78, 3.68, 3.58, 3.47, 3.37,\n", " 3.25, 3.14, 3.03])" ] } ], "prompt_number": 7 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### recarrays\n", "\n", "Have genfromtxt grab column names from the header and it returns a recarray. The recarray can be indexed numerically to get row data or with a column name to get a 1d array of data for that column." ] }, { "cell_type": "code", "collapsed": false, "input": [ "data = np.genfromtxt('BatteryParkTideData.csv', delimiter=',', names=True, missing='NA')" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 8 }, { "cell_type": "code", "collapsed": false, "input": [ "data" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 9, "text": [ "array([(0.0, 1.59, 4.68, 4.65), (0.1, 1.5, 4.55, 4.54),\n", " (0.2, 1.4, 4.46, 4.44), (0.3, 1.31, 4.36, 4.33),\n", " (0.4, 1.22, 4.28, 4.26), (0.5, 1.13, 4.21, 4.18),\n", " (0.6, 1.04, 4.15, 4.12), (0.7, 0.95, 4.08, 4.06),\n", " (0.8, 0.87, 3.99, 3.97), (0.9, 0.78, 3.92, 3.89),\n", " (1.0, 0.7, 3.87, 3.85), (1.1, 0.62, 3.86, 3.83),\n", " (1.2, 0.55, 3.8, 3.78), (1.3, 0.48, 3.74, 3.73),\n", " (1.4, 0.41, 3.68, 3.66), (1.5, 0.34, 3.63, 3.62),\n", " (1.6, 0.28, 3.59, 3.58), (1.7, 0.23, 3.55, 3.53),\n", " (1.8, 0.18, 3.5, 3.48), (1.9, 0.14, 3.45, 3.42),\n", " (2.0, 0.1, 3.39, 3.35), (2.1, 0.08, 3.37, 3.34),\n", " (2.2, 0.06, 3.33, 3.31), (2.3, 0.05, 3.31, 3.3),\n", " (2.4, 0.05, 3.29, 3.26), (2.5, 0.06, 3.25, 3.22),\n", " (2.6, 0.08, 3.21, 3.19), (2.7, 0.11, 3.19, 3.17),\n", " (2.8, 0.14, 3.19, 3.17), (2.9, 0.19, 3.2, 3.18),\n", " (3.0, 0.25, 3.23, 3.23), (3.1, 0.31, 3.29, 3.26),\n", " (3.2, 0.39, 3.31, 3.29), (3.3, 0.47, 3.34, 3.32),\n", " (3.4, 0.56, 3.39, 3.37), (3.5, 0.66, 3.44, 3.41),\n", " (3.6, 0.76, 3.49, 3.46), (3.7, 0.87, 3.57, 3.54),\n", " (3.8, 0.98, 3.67, 3.65), (3.9, 1.1, 3.78, 3.76),\n", " (4.0, 1.22, 3.87, 3.85), (4.1, 1.35, 3.96, 3.95),\n", " (4.2, 1.48, 4.09, 4.06), (4.3, 1.61, 4.19, 4.18),\n", " (4.4, 1.74, 4.36, 4.33), (4.5, 1.88, 4.49, 4.46),\n", " (4.6, 2.01, 4.59, 4.57), (4.7, 2.15, 4.68, 4.66),\n", " (4.8, 2.28, 4.79, 4.77), (4.9, 2.41, 4.9, 4.88),\n", " (5.0, 2.55, 5.01, 4.98), (5.1, 2.68, 5.12, 5.11),\n", " (5.2, 2.81, 5.26, 5.24), (5.3, 2.94, 5.38, 5.35),\n", " (5.4, 3.06, 5.52, 5.5), (5.5, 3.19, 5.69, 5.67),\n", " (5.6, 3.31, 5.84, 5.83), (5.7, 3.43, 5.97, 5.96),\n", " (5.8, 3.55, 6.13, 6.11), (5.9, 3.66, 6.26, 6.24),\n", " (6.0, 3.78, 6.39, 6.38), (6.1, 3.89, 6.53, 6.51),\n", " (6.2, 3.99, 6.68, 6.66), (6.3, 4.1, 6.84, 6.82),\n", " (6.4, 4.2, 7.0, 6.98), (6.5, 4.3, 7.13, 7.1),\n", " (6.6, 4.39, 7.25, 7.23), (6.7, 4.48, 7.36, 7.32),\n", " (6.8, 4.57, 7.46, 7.43), (6.9, 4.66, 7.56, 7.53),\n", " (7.0, 4.73, 7.65, 7.62), (7.1, 4.81, 7.71, 7.71),\n", " (7.2, 4.89, 7.8, 7.78), (7.3, 4.95, 7.9, 7.88),\n", " (7.4, 5.02, 8.02, 7.98), (7.5, 5.08, 8.07, 8.04),\n", " (7.6, 5.13, 8.12, 8.1), (7.7, 5.18, 8.26, 8.23),\n", " (7.8, 5.23, 8.36, 8.31), (7.9, 5.27, 8.47, 8.35),\n", " (8.0, 5.3, 8.53, 8.35), (8.1, 5.33, 8.58, 8.35),\n", " (8.2, 5.35, 8.58, 8.35), (8.3, 5.37, 8.6, 8.35),\n", " (8.4, 5.38, 8.67, 8.36), (8.5, 5.38, 8.69, 8.35),\n", " (8.6, 5.38, 8.67, 8.34), (8.7, 5.37, 8.71, 8.34),\n", " (8.8, 5.36, 8.76, 8.34), (8.9, 5.34, 8.79, 8.35),\n", " (9.0, 5.31, 8.8, 8.32), (9.1, 5.28, 8.85, 8.34),\n", " (9.2, 5.24, 8.85, 8.34), (9.3, 5.19, 8.85, 8.29),\n", " (9.4, 5.13, 8.85, 8.32), (9.5, 5.07, 8.81, 8.3),\n", " (9.6, 5.0, 8.78, 8.33), (9.7, 4.93, 8.69, 8.34),\n", " (9.8, 4.85, 8.6, 8.29), (9.9, 4.76, 8.58, 8.28),\n", " (10.0, 4.67, 8.53, 8.25), (10.1, 4.57, 8.5, 8.26),\n", " (10.2, 4.47, nan, nan), (10.3, 4.37, nan, nan),\n", " (10.4, 4.26, nan, nan), (10.5, 4.15, nan, nan),\n", " (10.6, 4.03, nan, nan), (10.7, 3.92, 8.1, 8.06),\n", " (10.8, 3.8, 8.05, 7.99), (10.9, 3.68, 7.94, 7.88),\n", " (11.0, 3.56, 7.83, 7.81), (11.1, 3.44, 7.77, 7.75),\n", " (11.2, 3.32, 7.68, 7.64), (11.3, 3.21, 7.56, 7.53),\n", " (11.4, 3.09, 7.45, 7.41), (11.5, 2.97, 7.33, 7.3),\n", " (11.6, 2.86, 7.22, 7.18), (11.7, 2.74, 7.09, 7.06),\n", " (11.8, 2.63, 6.96, 6.92), (11.9, 2.52, 6.84, 6.81),\n", " (12.0, 2.41, 6.77, 6.73), (12.1, 2.3, 6.65, 6.63),\n", " (12.2, 2.2, 6.58, 6.55), (12.3, 2.09, 6.5, 6.47),\n", " (12.4, 1.98, 6.44, 6.41), (12.5, 1.88, 6.33, 6.3),\n", " (12.6, 1.78, 6.24, 6.21), (12.7, 1.68, 6.16, 6.15),\n", " (12.8, 1.57, 6.1, 6.09), (12.9, 1.47, 6.02, 6.0),\n", " (13.0, 1.37, 5.91, 5.9), (13.1, 1.27, 5.87, 5.83),\n", " (13.2, 1.17, 5.79, 5.76), (13.3, 1.08, 5.69, 5.67),\n", " (13.4, 0.98, 5.61, 5.59), (13.5, 0.89, 5.55, 5.53),\n", " (13.6, 0.8, 5.46, 5.44), (13.7, 0.71, 5.42, 5.39),\n", " (13.8, 0.62, 5.37, 5.34), (13.9, 0.54, 5.33, 5.3),\n", " (14.0, 0.46, 5.3, 5.26), (14.1, 0.39, 5.27, 5.24),\n", " (14.2, 0.32, 5.26, 5.23), (14.3, 0.26, nan, 5.2),\n", " (14.4, 0.21, 5.27, 5.24), (14.5, 0.16, 5.28, 5.25),\n", " (14.6, 0.12, 5.29, 5.27), (14.7, 0.09, 5.38, 5.35),\n", " (14.8, 0.07, 5.43, 5.4), (14.9, 0.06, 5.52, 5.5),\n", " (15.0, 0.05, 5.6, 5.58), (15.1, 0.06, 5.69, 5.69),\n", " (15.2, 0.08, 5.79, 5.77), (15.3, 0.1, 5.97, 5.97),\n", " (15.4, 0.13, 6.11, 6.1), (15.5, 0.18, 6.24, 6.21),\n", " (15.6, 0.23, 6.4, 6.36), (15.7, 0.29, 6.5, 6.46),\n", " (15.8, 0.36, 6.61, 6.56), (15.9, 0.44, 6.7, 6.67),\n", " (16.0, 0.52, 6.85, 6.82), (16.1, 0.61, 7.02, 6.97),\n", " (16.2, 0.71, 7.15, 7.09), (16.3, 0.81, 7.28, 7.23),\n", " (16.4, 0.91, 7.45, 7.4), (16.5, 1.02, 7.57, 7.51),\n", " (16.6, 1.13, 7.75, 7.7), (16.7, 1.25, 7.91, 7.87),\n", " (16.8, 1.36, 8.03, 8.0), (16.9, 1.49, 8.18, 8.14),\n", " (17.0, 1.6, 8.27, 8.23), (17.1, 1.73, 8.38, 8.3),\n", " (17.2, 1.85, 8.48, 8.33), (17.3, 1.97, 8.63, 8.32),\n", " (17.4, 2.09, 8.77, 8.33), (17.5, 2.21, 8.9, 8.31),\n", " (17.6, 2.33, 9.03, 8.32), (17.7, 2.45, 9.19, 8.29),\n", " (17.8, 2.57, 9.33, 8.31), (17.9, 2.68, 9.48, 8.3),\n", " (18.0, 2.8, 9.62, 7.85), (18.1, 2.91, 9.74, 7.83),\n", " (18.2, 3.02, 9.95, 7.07), (18.3, 3.13, 10.1, 6.1),\n", " (18.4, 3.23, 10.22, 6.09), (18.5, 3.33, 10.39, 6.11),\n", " (18.6, 3.44, 10.55, 6.11), (18.7, 3.53, 10.69, 6.12),\n", " (18.8, 3.63, 10.88, 6.6), (18.9, 3.72, 11.07, 6.91),\n", " (19.0, 3.81, 11.25, 7.27), (19.1, 3.9, 11.41, 7.16),\n", " (19.2, 3.98, 11.62, 7.07), (19.3, 4.06, 11.87, 7.31),\n", " (19.4, 4.14, 12.09, 7.06), (19.5, 4.21, 12.33, 7.06),\n", " (19.6, 4.28, 12.54, 7.24), (19.7, 4.34, 12.75, 7.13),\n", " (19.8, 4.41, 12.93, 7.16), (19.9, 4.46, 13.04, 7.09),\n", " (20.0, 4.51, 13.15, 7.16), (20.1, 4.56, 13.2, 7.16),\n", " (20.2, 4.6, 13.26, 7.11), (20.3, 4.64, 13.34, 7.15),\n", " (20.4, 4.67, 13.4, 7.26), (20.5, 4.69, 13.46, 7.13),\n", " (20.6, 4.71, 13.54, 7.0), (20.7, 4.73, 13.65, 6.68),\n", " (20.8, 4.74, 13.72, 6.85), (20.9, 4.74, 13.78, 7.12),\n", " (21.0, 4.74, 13.81, 7.07), (21.1, 4.72, 13.85, 7.3),\n", " (21.2, 4.71, 13.87, 7.3), (21.3, 4.69, 13.87, 7.32),\n", " (21.4, 4.65, 13.88, 7.19), (21.5, 4.62, 13.79, 7.14),\n", " (21.6, 4.57, 13.72, 7.18), (21.7, 4.52, 13.63, 7.03),\n", " (21.8, 4.46, 13.54, 7.32), (21.9, 4.4, 13.41, 7.04),\n", " (22.0, 4.33, 13.3, 7.23), (22.1, 4.25, 13.15, 6.88),\n", " (22.2, 4.17, 12.99, 6.97), (22.3, 4.08, 12.86, 7.19),\n", " (22.4, 3.99, 12.69, 7.1), (22.5, 3.89, 12.5, 7.18),\n", " (22.6, 3.79, 12.27, 7.18), (22.7, 3.68, 12.07, 7.4),\n", " (22.8, 3.57, 11.87, 7.09), (22.9, 3.46, 11.61, 7.03),\n", " (23.0, 3.35, 11.32, 7.18), (23.1, 3.23, 11.04, 7.26),\n", " (23.2, 3.11, 10.78, 7.24), (23.3, 3.0, 10.47, 6.43),\n", " (23.4, 2.88, 10.15, 6.8), (23.5, 2.76, 9.81, 7.67),\n", " (23.6, 2.65, 9.54, 8.14), (23.7, 2.53, 9.22, 8.31),\n", " (23.8, 2.42, 8.92, 8.32), (23.9, 2.31, 8.62, 8.29),\n", " (24.0, 2.2, 8.35, 8.17), (24.1, 2.1, 8.03, 7.94),\n", " (24.2, 1.99, 7.77, 7.71), (24.3, 1.89, 7.58, 7.53),\n", " (24.4, 1.79, 7.42, 7.36), (24.5, 1.7, 7.2, 7.19),\n", " (24.6, 1.6, 7.06, 7.0), (24.7, 1.51, 6.87, 6.84),\n", " (24.8, 1.42, 6.71, 6.68), (24.9, 1.33, 6.54, 6.51),\n", " (25.0, 1.25, 6.4, 6.36), (25.1, 1.16, 6.26, 6.22),\n", " (25.2, 1.08, 6.13, 6.09), (25.3, 1.0, 5.99, 5.96),\n", " (25.4, 0.92, 5.85, 5.84), (25.5, 0.85, 5.77, 5.74),\n", " (25.6, 0.77, 5.64, 5.62), (25.7, 0.7, 5.52, 5.51),\n", " (25.8, 0.63, 5.36, 5.35), (25.9, 0.56, 5.21, 5.2),\n", " (26.0, 0.5, 5.08, 5.08), (26.1, 0.44, 4.94, 4.94),\n", " (26.2, 0.38, 4.8, 4.8), (26.3, 0.33, 4.7, 4.67),\n", " (26.4, 0.29, 4.55, 4.53), (26.5, 0.24, 4.44, 4.41),\n", " (26.6, 0.21, 4.32, 4.31), (26.7, 0.18, 4.21, 4.2),\n", " (26.8, 0.16, 4.1, 4.08), (26.9, 0.15, 4.0, 3.96),\n", " (27.0, 0.14, 3.91, 3.87), (27.1, 0.14, 3.81, 3.79),\n", " (27.2, 0.16, 3.77, 3.75), (27.3, 0.18, 3.71, 3.68),\n", " (27.4, 0.21, 3.67, 3.63), (27.5, 0.25, 3.64, 3.61),\n", " (27.6, 0.3, 3.63, 3.61), (27.7, 0.36, 3.67, 3.64),\n", " (27.8, 0.43, 3.69, 3.67), (27.9, 0.5, 3.72, 3.71),\n", " (28.0, 0.59, 3.81, 3.79), (28.1, 0.68, 3.88, 3.87),\n", " (28.2, 0.78, 4.0, 3.99), (28.3, 0.89, 4.07, 4.05),\n", " (28.4, 1.0, 4.14, 4.14), (28.5, 1.12, 4.21, 4.2),\n", " (28.6, 1.24, 4.3, 4.3), (28.7, 1.37, 4.41, 4.41),\n", " (28.8, 1.5, 4.52, 4.53), (28.9, 1.63, 4.65, 4.66),\n", " (29.0, 1.77, 4.79, 4.81), (29.1, 1.91, 4.95, 4.97),\n", " (29.2, 2.05, 5.08, 5.13), (29.3, 2.19, 5.21, 5.28),\n", " (29.4, 2.32, 5.36, 5.44), (29.5, 2.46, 5.5, 5.57),\n", " (29.6, 2.6, 5.6, 5.71), (29.7, 2.73, 5.75, 5.87),\n", " (29.8, 2.87, 5.89, 6.0), (29.9, 3.0, 5.99, 6.11),\n", " (30.0, 3.13, 6.09, 6.21), (30.1, 3.25, 6.17, 6.27),\n", " (30.2, 3.38, 6.27, 6.4), (30.3, 3.5, 6.37, 6.5),\n", " (30.4, 3.62, 6.43, 6.54), (30.5, 3.73, 6.51, 6.62),\n", " (30.6, 3.84, 6.56, 6.7), (30.7, 3.95, 6.61, 6.75),\n", " (30.8, 4.06, 6.7, 6.8), (30.9, 4.16, 6.74, 6.86),\n", " (31.0, 4.25, 6.81, 6.93), (31.1, 4.35, 6.84, 6.98),\n", " (31.2, 4.44, 6.92, 7.04), (31.3, 4.52, 6.93, 7.07),\n", " (31.4, 4.6, 6.93, 7.09), (31.5, 4.68, 6.96, 7.09),\n", " (31.6, 4.76, 6.95, 7.1), (31.7, 4.83, 6.96, 7.12),\n", " (31.8, 4.89, 6.97, 7.12), (31.9, 4.95, 7.01, 7.15),\n", " (32.0, 5.01, 7.02, 7.15), (32.1, 5.07, 7.02, 7.15),\n", " (32.2, 5.11, 7.03, 7.16), (32.3, 5.16, 7.02, 7.16),\n", " (32.4, 5.2, 7.01, 7.15), (32.5, 5.23, 7.03, 7.17),\n", " (32.6, 5.26, 7.05, 7.21), (32.7, 5.29, 7.13, 7.28),\n", " (32.8, 5.31, 7.18, 7.32), (32.9, 5.32, 7.19, 7.34),\n", " (33.0, 5.33, 7.17, 7.31), (33.1, 5.33, 7.13, 7.26),\n", " (33.2, 5.33, 7.11, 7.24), (33.3, 5.32, 7.12, 7.26),\n", " (33.4, 5.31, 7.12, 7.26), (33.5, 5.29, 7.11, 7.25),\n", " (33.6, 5.26, 7.12, 7.25), (33.7, 5.23, 7.16, 7.3),\n", " (33.8, 5.19, 7.2, 7.36), (33.9, 5.14, 7.24, 7.38),\n", " (34.0, 5.1, 7.28, 7.41), (34.1, 5.04, 7.29, 7.44),\n", " (34.2, 4.97, 7.33, 7.48), (34.3, 4.9, 7.31, 7.47),\n", " (34.4, 4.83, 7.27, 7.42), (34.5, 4.75, 7.24, 7.4),\n", " (34.6, 4.66, 7.19, 7.34), (34.7, 4.57, 7.11, 7.25),\n", " (34.8, 4.47, 6.99, 7.14), (34.9, 4.37, 6.87, 7.01),\n", " (35.0, 4.27, 6.72, 6.88), (35.1, 4.16, 6.64, 6.79),\n", " (35.2, 4.05, 6.56, 6.69), (35.3, 3.93, 6.46, 6.61),\n", " (35.4, 3.81, 6.37, 6.53), (35.5, 3.69, 6.25, 6.41),\n", " (35.6, 3.57, 6.11, 6.28), (35.7, 3.45, 6.01, 6.18),\n", " (35.8, 3.33, 5.91, 6.07), (35.9, 3.21, 5.78, 5.98),\n", " (36.0, 3.09, 5.66, 5.83), (36.1, 2.97, 5.49, 5.64),\n", " (36.2, 2.86, 5.31, 5.49), (36.3, 2.74, 5.15, 5.29),\n", " (36.4, 2.63, 5.03, 5.16), (36.5, 2.51, 4.91, 5.02),\n", " (36.6, 2.4, 4.78, 4.9), (36.7, 2.29, 4.64, 4.74),\n", " (36.8, 2.19, 4.51, 4.59), (36.9, 2.08, 4.36, 4.44),\n", " (37.0, 1.98, 4.21, 4.29), (37.1, 1.88, 4.05, 4.11),\n", " (37.2, 1.78, 3.91, 3.96), (37.3, 1.68, 3.76, 3.81),\n", " (37.4, 1.58, 3.63, 3.68), (37.5, 1.49, 3.51, 3.56),\n", " (37.6, 1.39, 3.4, 3.43), (37.7, 1.3, 3.3, 3.34),\n", " (37.8, 1.21, 3.21, 3.23), (37.9, 1.12, 3.08, 3.09),\n", " (38.0, 1.03, 2.94, 2.96), (38.1, 0.94, 2.83, 2.83),\n", " (38.2, 0.85, 2.73, 2.72), (38.3, 0.77, 2.62, 2.61),\n", " (38.4, 0.69, 2.51, 2.49), (38.5, 0.6, 2.4, 2.39),\n", " (38.6, 0.53, 2.34, 2.31), (38.7, 0.46, 2.23, 2.21),\n", " (38.8, 0.39, 2.14, 2.11), (38.9, 0.33, 2.03, 2.0),\n", " (39.0, 0.27, 1.92, 1.89), (39.1, 0.22, 1.84, 1.83),\n", " (39.2, 0.18, 1.77, 1.77), (39.3, 0.14, 1.71, 1.69),\n", " (39.4, 0.11, 1.67, 1.65), (39.5, 0.09, 1.64, 1.61),\n", " (39.6, 0.08, 1.6, 1.58), (39.7, 0.08, 1.58, 1.56),\n", " (39.8, 0.09, 1.58, 1.54), (39.9, 0.1, 1.56, 1.52),\n", " (40.0, 0.13, 1.53, 1.51), (40.1, 0.16, 1.52, 1.51),\n", " (40.2, 0.2, 1.53, 1.52), (40.3, 0.26, 1.55, 1.53),\n", " (40.4, 0.32, 1.54, 1.53), (40.5, 0.39, 1.58, 1.57),\n", " (40.6, 0.47, 1.64, 1.64), (40.7, 0.55, 1.7, 1.7),\n", " (40.8, 0.64, 1.83, 1.8), (40.9, 0.74, 1.94, 1.93),\n", " (41.0, 0.84, 2.03, 2.02), (41.1, 0.95, 2.14, 2.13),\n", " (41.2, 1.06, 2.26, 2.26), (41.3, 1.17, 2.4, 2.41),\n", " (41.4, 1.29, 2.57, 2.58), (41.5, 1.41, 2.73, 2.73),\n", " (41.6, 1.53, 2.89, 2.89), (41.7, 1.66, 3.03, 3.04),\n", " (41.8, 1.78, 3.17, 3.2), (41.9, 1.9, 3.33, 3.37),\n", " (42.0, 2.02, 3.5, 3.54), (42.1, 2.15, 3.66, 3.71),\n", " (42.2, 2.27, 3.83, 3.9), (42.3, 2.39, 4.01, 4.08),\n", " (42.4, 2.5, 4.17, 4.25), (42.5, 2.62, 4.29, 4.4),\n", " (42.6, 2.73, 4.44, 4.55), (42.7, 2.84, 4.6, 4.71),\n", " (42.8, 2.95, 4.78, 4.88), (42.9, 3.05, 4.92, 5.03),\n", " (43.0, 3.16, 4.99, 5.13), (43.1, 3.26, 5.1, 5.26),\n", " (43.2, 3.36, 5.22, 5.38), (43.3, 3.45, 5.35, 5.5),\n", " (43.4, 3.54, 5.41, 5.58), (43.5, 3.63, 5.5, 5.69),\n", " (43.6, 3.71, 5.58, 5.76), (43.7, 3.8, 5.64, 5.83),\n", " (43.8, 3.87, 5.68, 5.87), (43.9, 3.95, 5.73, 5.93),\n", " (44.0, 4.02, 5.79, 5.98), (44.1, 4.09, 5.79, 6.0),\n", " (44.2, 4.15, 5.81, 6.02), (44.3, 4.22, 5.83, 6.03),\n", " (44.4, 4.27, 5.86, 6.06), (44.5, 4.32, 5.88, 6.08),\n", " (44.6, 4.37, 5.89, 6.09), (44.7, 4.42, 5.88, 6.08),\n", " (44.8, 4.46, 5.88, 6.07), (44.9, 4.49, 5.88, 6.07),\n", " (45.0, 4.52, 5.88, 6.08), (45.1, 4.55, 5.91, 6.1),\n", " (45.2, 4.57, 5.91, 6.1), (45.3, 4.58, 5.9, 6.1),\n", " (45.4, 4.6, 5.9, 6.1), (45.5, 4.6, 5.87, 6.07),\n", " (45.6, 4.6, 5.84, 6.05), (45.7, 4.59, 5.8, 6.01),\n", " (45.8, 4.58, 5.79, 6.0), (45.9, 4.56, 5.77, 5.97),\n", " (46.0, 4.54, 5.72, 5.93), (46.1, 4.51, 5.68, 5.89),\n", " (46.2, 4.48, 5.64, 5.84), (46.3, 4.43, 5.58, 5.78),\n", " (46.4, 4.38, 5.51, 5.71), (46.5, 4.33, 5.44, 5.63),\n", " (46.6, 4.27, 5.38, 5.57), (46.7, 4.2, 5.32, 5.5),\n", " (46.8, 4.13, 5.23, 5.41), (46.9, 4.05, 5.15, 5.32),\n", " (47.0, 3.96, 5.07, 5.24), (47.1, 3.87, 4.96, 5.13),\n", " (47.2, 3.78, 4.86, 5.03), (47.3, 3.68, 4.75, 4.92),\n", " (47.4, 3.58, 4.63, 4.8), (47.5, 3.47, 4.52, 4.69),\n", " (47.6, 3.37, 4.42, 4.6), (47.7, 3.25, 4.32, 4.5),\n", " (47.8, 3.14, 4.22, 4.39), (47.9, 3.03, 4.12, 4.28)], \n", " dtype=[('TimeOffsetHours', ' 5]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 25, "text": [ "array([ 5.02, 5.08, 5.13, 5.18, 5.23, 5.27, 5.3 , 5.33, 5.35,\n", " 5.37, 5.38, 5.38, 5.38, 5.37, 5.36, 5.34, 5.31, 5.28,\n", " 5.24, 5.19, 5.13, 5.07, 5.01, 5.07, 5.11, 5.16, 5.2 ,\n", " 5.23, 5.26, 5.29, 5.31, 5.32, 5.33, 5.33, 5.33, 5.32,\n", " 5.31, 5.29, 5.26, 5.23, 5.19, 5.14, 5.1 , 5.04])" ] } ], "prompt_number": 25 }, { "cell_type": "code", "collapsed": false, "input": [ "pred[(pred > 5) | (pred < 0.5)]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 26, "text": [ "array([ 0.48, 0.41, 0.34, 0.28, 0.23, 0.18, 0.14, 0.1 , 0.08,\n", " 0.06, 0.05, 0.05, 0.06, 0.08, 0.11, 0.14, 0.19, 0.25,\n", " 0.31, 0.39, 0.47, 5.02, 5.08, 5.13, 5.18, 5.23, 5.27,\n", " 5.3 , 5.33, 5.35, 5.37, 5.38, 5.38, 5.38, 5.37, 5.36,\n", " 5.34, 5.31, 5.28, 5.24, 5.19, 5.13, 5.07, 0.46, 0.39,\n", " 0.32, 0.26, 0.21, 0.16, 0.12, 0.09, 0.07, 0.06, 0.05,\n", " 0.06, 0.08, 0.1 , 0.13, 0.18, 0.23, 0.29, 0.36, 0.44,\n", " 0.44, 0.38, 0.33, 0.29, 0.24, 0.21, 0.18, 0.16, 0.15,\n", " 0.14, 0.14, 0.16, 0.18, 0.21, 0.25, 0.3 , 0.36, 0.43,\n", " 5.01, 5.07, 5.11, 5.16, 5.2 , 5.23, 5.26, 5.29, 5.31,\n", " 5.32, 5.33, 5.33, 5.33, 5.32, 5.31, 5.29, 5.26, 5.23,\n", " 5.19, 5.14, 5.1 , 5.04, 0.46, 0.39, 0.33, 0.27, 0.22,\n", " 0.18, 0.14, 0.11, 0.09, 0.08, 0.08, 0.09, 0.1 , 0.13,\n", " 0.16, 0.2 , 0.26, 0.32, 0.39, 0.47])" ] } ], "prompt_number": 26 }, { "cell_type": "markdown", "metadata": {}, "source": [ "How exactly does this work? The truthy expressions with arrays produce another array: an array of boolean values with the same shape as the original array with `True` where the expression is true, and `False` elsewhere. Let's see how that looks on a small array:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "np.arange(10) > 5" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 27, "text": [ "array([False, False, False, False, False, False, True, True, True, True], dtype=bool)" ] } ], "prompt_number": 27 }, { "cell_type": "markdown", "metadata": {}, "source": [ "These boolean arrays can be saved in their own variables, combined logically with other boolean arrays, and used to index any array with the same shape.\n", "\n", "To get the indices where a condition is true use the [numpy.where](http://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html) function:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "np.where(np.arange(10) > 5)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 28, "text": [ "(array([6, 7, 8, 9]),)" ] } ], "prompt_number": 28 }, { "cell_type": "markdown", "metadata": {}, "source": [ "How does this help with the `nan` issue? Much like Python's standard library has a [math.isnan](http://docs.python.org/2/library/math.html#math.isnan) function that works on floats, there is a [numpy.isnan](http://docs.scipy.org/doc/numpy/reference/generated/numpy.isnan.html) function that works on arrays. (In fact, NumPy has array equivalents to most of the functions in the `math` module.) Here's a small example of `np.isnan`:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "a = np.array([1, 2, np.nan, 4, 5, np.nan])\n", "np.isnan(a)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 29, "text": [ "array([False, False, True, False, False, True], dtype=bool)" ] } ], "prompt_number": 29 }, { "cell_type": "markdown", "metadata": {}, "source": [ "`np.isnan` returns an array of booleans just like the logical expressions up above, so that looks promising! Let's try it:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "a[np.isnan(a)]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 30, "text": [ "array([ nan, nan])" ] } ], "prompt_number": 30 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course, that grabbed the `nan` values because `np.isnan` gives `True` where `a` has `nan` values. One thing to do perform a logical flip on the boolean array using the `~` operator:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "a[~np.isnan(a)]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 31, "text": [ "array([ 1., 2., 4., 5.])" ] } ], "prompt_number": 31 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or we could see what other kinds of [logical functions](http://docs.scipy.org/doc/numpy/reference/routines.logic.html) there are in NumPy. One is [numpy.isfinite](http://docs.scipy.org/doc/numpy/reference/generated/numpy.isfinite.html#numpy.isfinite):" ] }, { "cell_type": "code", "collapsed": false, "input": [ "a[np.isfinite(a)]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 32, "text": [ "array([ 1., 2., 4., 5.])" ] } ], "prompt_number": 32 }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can use these functions, along with the [any](http://docs.scipy.org/doc/numpy/reference/generated/numpy.any.html#numpy.any) or [all](http://docs.scipy.org/doc/numpy/reference/generated/numpy.all.html#numpy.all) fuctions/methods on arrays, to test for the presence of missing data:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# are there any nan values?\n", "print 'time:', np.isnan(time).any()\n", "print 'backup:', np.isnan(backup).any()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "time: False\n", "backup: True\n" ] } ], "prompt_number": 33 }, { "cell_type": "code", "collapsed": false, "input": [ "# are all of the values finite?\n", "print 'pred:', np.isfinite(pred).all()\n", "print 'accoustic:', np.isfinite(accoustic).all()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "pred: True\n", "accoustic: False\n" ] } ], "prompt_number": 34 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Both `backup` and `accoustic` have missing data. (These are the two columns of actual instrument measurements so it shouldn't be too shocking to see missing data.) We can use logical comparisons with arrays to make a boolean array of where `backup` and `accoustic` are both good:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "not_nan = np.isfinite(backup) & np.isfinite(accoustic)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 35 }, { "cell_type": "markdown", "metadata": {}, "source": [ "And then use this to make new copies of `time`, `pred`, `backup`, and `accoustic` without the rows where `backup` and `accoustic` are missing data:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "time = time[not_nan]\n", "pred = pred[not_nan]\n", "backup = backup[not_nan]\n", "accoustic = accoustic[not_nan]" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 36 }, { "cell_type": "markdown", "metadata": {}, "source": [ "How many rows did we lose?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "not_nan.size - time.size" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 37, "text": [ "6" ] } ], "prompt_number": 37 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not too bad. Now we can get down to business!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Analysis (and Plots)\n", "\n", "So what are we actually looking at here?\n", "\n", "* `time`: time in hours since the first measurement in the file\n", "* `pred`: predicted water level\n", "* `accoustic`: a measured water level\n", "* `backup`: another measured water level\n", "\n", "All water levels are in feet above [Mean Lower Low Water](http://tidesandcurrents.noaa.gov/mllw.html). Let's take a quick look:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print time[:5]\n", "print pred[:5]\n", "print accoustic[:5]\n", "print backup[:5]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[ 0. 0.1 0.2 0.3 0.4]\n", "[ 1.59 1.5 1.4 1.31 1.22]\n", "[ 4.65 4.54 4.44 4.33 4.26]\n", "[ 4.68 4.55 4.46 4.36 4.28]\n" ] } ], "prompt_number": 38 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Honestly, looking at a ton of numbers is a great way to get a feal for things. We could compare maxima:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "backup.max() - pred.max()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 39, "text": [ "8.5" ] } ], "prompt_number": 39 }, { "cell_type": "markdown", "metadata": {}, "source": [ "But are those at the same time? We can use the [argmax method](http://docs.scipy.org/doc/numpy/reference/generated/numpy.argmax.html) to get the index of the maxima of one and use that index in the other for a more apples-to-apples comparison:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "m = backup.argmax()\n", "print m\n", "backup[m] - pred[m]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "208\n" ] }, { "output_type": "pyout", "prompt_number": 40, "text": [ "9.2300000000000004" ] } ], "prompt_number": 40 }, { "cell_type": "markdown", "metadata": {}, "source": [ "So at least according to the `backup` measurements the MLLW was 9 feet higher than predicted at one point during Hurricane Sandy!\n", "\n", "But that's just one data point. To really see trends you want a plot. We'll start by turning on the IPython Notebook's inline plotting mode so that plots show up right here in our notebook:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "%pylab inline" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", "Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].\n", "For more information, type 'help(pylab)'.\n" ] } ], "prompt_number": 41 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then conigure plots to display as SVG (default is PNG):" ] }, { "cell_type": "code", "collapsed": false, "input": [ "%config InlineBackend.figure_format = 'svg'" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 42 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then we'll import [matplotlib](http://matplotlib.org):" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import matplotlib.pyplot as plt" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 43 }, { "cell_type": "markdown", "metadata": {}, "source": [ "And make a basic plot of our data, using `time` along the x-axis and plotting the predicted and measured levels as three separate lines:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "fig, ax = plt.subplots()\n", "ax.plot(time, pred)\n", "ax.plot(time, accoustic)\n", "ax.plot(time, backup)\n", "ax.set_ylabel('Feet above MLLW')\n", "ax.set_xlabel('Hours Since First Measurement')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 44, "text": [ "" ] }, { "output_type": "display_data", "svg": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ] } ], "prompt_number": 44 }, { "cell_type": "markdown", "metadata": {}, "source": [ "And to keep the lines straight we can throw in a legend:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "fig, ax = plt.subplots()\n", "ax.plot(time, pred, label='Predicted')\n", "ax.plot(time, accoustic, label='Accoustic')\n", "ax.plot(time, backup, label='Backup')\n", "ax.set_ylabel('Feet above MLLW')\n", "ax.set_xlabel('Hours Since First Measurement')\n", "ax.legend(loc='upper right')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 45, "text": [ "" ] }, { "output_type": "display_data", "svg": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ] } ], "prompt_number": 45 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Great! We can see that maybe something got a bit weird with the `accoustic` measurements and maybe we should trust the `backup` measurements more. Now maybe we'd like to quantify and plot the difference between the measured and predicted tide levels. Piece of cake:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "obs_minus_pred = backup - pred" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 46 }, { "cell_type": "code", "collapsed": false, "input": [ "fig, ax = plt.subplots()\n", "ax.plot(time, pred, label='Predicted')\n", "ax.plot(time, backup, label='Backup')\n", "ax.plot(time, obs_minus_pred, label='Difference')\n", "ax.set_ylabel('Feet above MLLW')\n", "ax.set_xlabel('Hours Since First Measurement')\n", "ax.legend(loc='upper right')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 47, "text": [ "" ] }, { "output_type": "display_data", "svg": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ] } ], "prompt_number": 47 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Wait a minute, what did we just do there? We just used a minus sign to do an element-wise subtraction of two arrays! Pretty handy. Let's look at array arithmetic for a bit." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Array Math\n", "\n", "Arrays can be used in arithmetic expressions using the same binary operators we use for numbers: `+`, `-`, `*`, `/`, `**`, etc. These expressions return new arrays in which the mathematical operation has been applied elementwise. It's easiest to see this when combining an array and a scalar:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "a = np.arange(5, dtype=np.float) # float to avoid integer surprises\n", "print a" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[ 0. 1. 2. 3. 4.]\n" ] } ], "prompt_number": 61 }, { "cell_type": "code", "collapsed": false, "input": [ "a + 5" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 62, "text": [ "array([ 5., 6., 7., 8., 9.])" ] } ], "prompt_number": 62 }, { "cell_type": "code", "collapsed": false, "input": [ "a * 5" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 63, "text": [ "array([ 0., 5., 10., 15., 20.])" ] } ], "prompt_number": 63 }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get this same effect with lists you'd need to use a list or comprehension. With arrays it's as simple as `a + 5`. When combining arrays and scalars the same operation is applied to every element of the array. When combining two arrays it's slightly different:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "b = np.arange(10, 20, 2, dtype=np.float)\n", "print b" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[ 10. 12. 14. 16. 18.]\n" ] } ], "prompt_number": 64 }, { "cell_type": "code", "collapsed": false, "input": [ "b - a" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 65, "text": [ "array([ 10., 11., 12., 13., 14.])" ] } ], "prompt_number": 65 }, { "cell_type": "code", "collapsed": false, "input": [ "a / b" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 66, "text": [ "array([ 0. , 0.08333333, 0.14285714, 0.1875 , 0.22222222])" ] } ], "prompt_number": 66 }, { "cell_type": "markdown", "metadata": {}, "source": [ "In these cases the first element of `a` operates with the first element of `b`, and so on. So long as the two arrays are the same size and shape you will see this behavior. Arrays with different shapes can sometimes be combined with binary operators via an implicit resizing/reshaping called [broadcasting](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html), but that's a topic for another day." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## More Data Analysis\n", "\n", "There might be more we could do with this data. When was the peak water height?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "time[backup.argmax()]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 81, "text": [ "21.399999999999999" ] } ], "prompt_number": 81 }, { "cell_type": "markdown", "metadata": {}, "source": [ "These are hours past midnight on Oct 29, so the peak water height was sometime around 9:24 PM on Oct. 29. The biggest difference between the predicted and observed tide heights were around the same time:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "time[obs_minus_pred.argmax()]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 79, "text": [ "21.399999999999999" ] } ], "prompt_number": 79 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Which is one reason NYC had such bad flooding." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Learn More\n", "\n", "These are just some ways of working with data in NumPy arrays. Learn more by diving into the documentation at http://docs.scipy.org/doc/." ] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] } ], "metadata": {} } ] }