From tick by tick data to candlestick

Question

I've tick by tick data for Forex pairs

Here is a sample of EURUSD/EURUSD-2012-06.csv

EUR/USD,20120601 00:00:00.207,1.23618,1.2363
EUR/USD,20120601 00:00:00.209,1.23618,1.23631
EUR/USD,20120601 00:00:00.210,1.23618,1.23631
EUR/USD,20120601 00:00:00.211,1.23623,1.23631
EUR/USD,20120601 00:00:00.240,1.23623,1.23627
EUR/USD,20120601 00:00:00.423,1.23622,1.23627
EUR/USD,20120601 00:00:00.457,1.2362,1.23626
EUR/USD,20120601 00:00:01.537,1.2362,1.23625
EUR/USD,20120601 00:00:03.010,1.2362,1.23624
EUR/USD,20120601 00:00:03.012,1.2362,1.23625

Full tick data can be downloaded here http://dl.free.fr/k4vVF7aOD

Columns are :

Symbol,Datetime,Bid,Ask

I would like to convert this tick by tick data to candlestick data (also called OHLC Open High Low Close) I will say that I want to get a M15 timeframe (15 minutes) as an example

I would like to use Python and Pandas library to achieve this task.

I've done a little part of the job... reading the tick by tick data file

Here is the code

#!/usr/bin/env python

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.finance import candlestick
from datetime import *

def conv_str_to_datetime(x):
    return(datetime.strptime(x, '%Y%m%d %H:%M:%S.%f'))

df = pd.read_csv('test_EURUSD/EURUSD-2012-07.csv', names=['Symbol', 'Date_Time', 'Bid', 'Ask'], converters={'Date_Time': conv_str_to_datetime})

PipPosition = 4
df['Spread'] = (df['Ask'] - df['Bid']) * 10**PipPosition

print(df)

print("="*10)

print(df.ix[0])

but now I don't know how to start rest of the job...

I want to get data like

Symbol,Datetime_open_candle,open_price,high_price,low_price,close_price

Price on candle will be based on Bid column.

The first part of the problem is in my mind to get the first Datetime_open_candle (compatible with the desired timeframe, lets say that the name of the variable is dt1) and the last Datetime_open_candle (let's say that the name of this variable is dt2).

After I will probably need to get data from dt1 to dt2 (and not data before dt1 and after dt2)

Knowing dt1 and dt2 and desired timeframe I can know the number of candles I will have...

I've "just to" know, for each candle, what is open/high/low/close price.

I'm looking for a quite fast algorithm, if possible a vectorized one (if it's possible) as tick data can be very big.

Answer 1

In [59]: df
Out[59]:
                             Symbol      Bid      Ask
Datetime
2012-06-01 00:00:00.207000  EUR/USD  1.23618  1.23630
2012-06-01 00:00:00.209000  EUR/USD  1.23618  1.23631
2012-06-01 00:00:00.210000  EUR/USD  1.23618  1.23631
2012-06-01 00:00:00.211000  EUR/USD  1.23623  1.23631
2012-06-01 00:00:00.240000  EUR/USD  1.23623  1.23627
2012-06-01 00:00:00.423000  EUR/USD  1.23622  1.23627
2012-06-01 00:00:00.457000  EUR/USD  1.23620  1.23626
2012-06-01 00:00:01.537000  EUR/USD  1.23620  1.23625
2012-06-01 00:00:03.010000  EUR/USD  1.23620  1.23624
2012-06-01 00:00:03.012000  EUR/USD  1.23620  1.23625

In [60]: grouped = df.groupby('Symbol')

In [61]: ask =  grouped['Ask'].resample('15Min', how='ohlc')

In [62]: bid = grouped['Bid'].resample('15Min', how='ohlc')

In [63]: pandas.concat([ask, bid], axis=1, keys=['Ask', 'Bid'])
Out[63]:
                                Ask                                 Bid
                               open     high      low    close     open     high      low   close
Symbol  Datetime
EUR/USD 2012-06-01 00:15:00  1.2363  1.23631  1.23624  1.23625  1.23618  1.23623  1.23618  1.2362

Answer 2

The syntax in the answer from Overmeire is meanwhile deprecated.

Instead of this:

ask =  grouped['Ask'].resample('15Min', how='ohlc')
bid = grouped['Bid'].resample('15Min', how='ohlc')

Use this:

ask =  grouped['Ask'].resample('15Min').ohlc()
bid = grouped['Bid'].resample('15Min').ohlc()