Groovy over Python?

After a few frustrating events where I had some python code blow up because of dependencies, I started looking hard a using Groovy going forward.

For some simple things, Groovy and Python are very easy. For example if I wanted to read the latest sales from Park.IO and print them out, I could do the following in Python on my Mac.

import pandas as pd

df = pd.read_csv('https://park.io/orders/export.csv')

print (df)

If I tried that on my old Raspberry Pi, I'd run into dependency issues w.r.t. to Numpy not compiling correctly.

With Groovy, it's roughly the same but no need to call a module, it's just built in.

String getResult = new URL('https://park.io/orders/export.csv').text

print getResult

And that works on my Raspberry Pi.

Granted, I don't really need all the multi-threading power of Java to run this simple program but the portability is nice.

comments

Groovy for Data Science

I was looking through Reddit the other day and came across an interesting post in r/Groovy. The group itself has been relatively dead for a while but sprang to life with a question about using Groovy for Data Science. Granted, Python IS eating the world right now for Data Science with R right behind it and Java still hanging out with all the Big Data stuff.

I've always liked Groovy because it's very Python like (high level-ish) but compiles down to byte-code during runtime. It's multithreaded and concurrent. I know that Python attempts to do this and there are many hackish ways to make it as fast as C, but you have to know how to do it. High performance python is an art form beyond the Data Science world IMHO.

The Reddit post had a link to a set of slides from Dr. Paul King of Object Computing Inc. I did some sleuthing and came across an hour long presentation and I think it's fantastic.

comments

The Contrarian Case for Getting Active

Right before the New Year I posted an article on Medium titled "Is Passive Investing Going to Kill Us?" It centers around one particular chart where the amount of money flowing into passive investing is staggering.

While I'm a big fan of Passive Investing and I've been riding its success, I can't help but wonder if I was just plain lucky. I know that markets love to correct imbalances but the question is when? Perhaps it might be prudent to think of active strategies again.

Enter Python

I've been using the heck out of Python lately and I discovered a Forex library (it's V20 compatible) that lets me connect to Oanda easily. I did some more research and found a simple momentum trading bot in about 100 lines of python. Of course, this blog post was old and it didn't work out of the box. On top of that it used a deprecated library to connect to Oanda. Luckily I was able to find a brave soul that did a lot of the updates to get it 90% of the way there.

I got the sucker up and running and wouldn't you know it, it sucks. It loses a money when I trade it using a 'paper trading' account. When you think about it, this is the right way of doing it. Test it out using a paper trading account and see how it performs.

Trading Bot Building Process

A customer of mine suggested reading "Advances in Financial Machine Learning" by Marcos Lopez de Prado. It's quite a heavy read but Marcos shares A TON of good stuff in there. Most importantly in my opinion is how to build these trading bots or systems starting in Chapter 1.

It all starts with the Data Curators, the ones that pull in all the tick data and other stuff. They're the ones that augment, compile, aggregate, and curate the data for downstream use. This is a crucial step, if this is not done well, your models will suffer.

Next it's the Feature Analysts. They're the ones that try to extract interesting features from the curated data set. This is also a crucial step because in my experience, good features can make you and bad features can break you.

The third step is where the Strategists come in, they are the ones that are responsible for taking the Data and Features and devising a strategy to use in the markets. When they've come up with a model, they pass this off to the next group, the Back Testers. The Back Testers do just that, they take the new strategy and test it in various market conditions to see if it actually produces alpha.

If the new strategy produces alpha, then the Back Testers move this model over to the Deployment Team. This team is responsible for putting it into production, whether it uses CPU or GPU power, how it's called, etc. The Deployment Team is also responsible for taking this strategy OUT of production once the market has figured out this strategy and you're no longer making money with it.

The last step is Portfolio Oversight. This is where the back tested and deployed strategy is carefully monitored. First from a 'paper trading' implementation, then to live trading. This is where it should start producing returns and where it gets taken out if the returns start faltering.

The book is full of more great information but as I wrote above, it's not for the feint hearted. It's invaluable IMHO if you want to build Trading Bots in a serious manner.

The Python Code

Without much further ado, here's my cobbled together Momentum Trading Bot. Note, the code is not elegant as I'm still working through stuff and it's very simplistic. Works with Python 3.6+ and you'll need to add an account.txt file (the account number from Oanda) and a token.txt file (the V20 API token from Oanda).

#!/usr/bin/env python
# coding: utf-8

# In[1]:


import json
import oandapyV20 as opy
import oandapyV20.endpoints.instruments as instruments
from oandapyV20.contrib.factories import InstrumentsCandlesFactory

import pandas as pd
from pandas.io.json import json_normalize

from oandapyV20.exceptions import V20Error, StreamTerminated
from oandapyV20.endpoints.transactions import TransactionsStream
from oandapyV20.endpoints.pricing import PricingStream
from oandapyV20.contrib.requests import TrailingStopLossOrderRequest

import datetime
from dateutil import parser

import numpy as np


# In[2]:


def exampleAuth():
    accountID, token = None, None
    with open("./oanda_account/account.txt") as I:
        accountID = I.read().strip()
    with open("./oanda_account/token.txt") as I:
        token = I.read().strip()
    return accountID, token


# In[3]:


instrument = "AUD_USD"

#Set time functions to offset chart
today = datetime.datetime.today()
two_years_ago = today - datetime.timedelta(days=720)

current_time = datetime.datetime.now()
twentyfour_hours_ago = current_time - datetime.timedelta(days=2)

print (current_time)
print (twentyfour_hours_ago)


# In[4]:


#Create time parameter for Oanada call
ct = current_time.strftime("%Y-%m-%dT%H:%M:%SZ")
tf = twentyfour_hours_ago.strftime("%Y-%m-%dT%H:%M:%SZ")


# In[5]:


#Connect to tokens
accountID, access_token = exampleAuth()
client = opy.API(access_token=access_token)


# In[6]:


# The v20 api handles from times a little differently - be careful of the timezone
params={"from": tf,
        "to": ct,
        "granularity":'M1',
        "price":'A'}
r = instruments.InstrumentsCandles(instrument=instrument,params=params)
#Do not use client from above
data = client.request(r)
results= [{"time":x['time'],"closeAsk":float(x['ask']['c'])} for x in data['candles']]
df = pd.DataFrame(results).set_index('time')

df.index = pd.DatetimeIndex(df.index)

df.info()
df.head()


# In[7]:


df['returns'] = np.log(df['closeAsk'] / df['closeAsk'].shift(1))

cols = []

for momentum in [15, 30, 60, 120]:
    col = 'position_%s' % momentum
    df[col] = np.sign(df['returns'].rolling(momentum).mean())
    cols.append(col)


# In[8]:


get_ipython().run_line_magic('matplotlib', 'inline')
import seaborn as sns; sns.set()

strats = ['returns']

for col in cols:
    strat = 'strategy_%s' % col.split('_')[1] 
    df[strat] = df[col].shift(1) * df['returns']
    strats.append(strat)


# In[9]:


df[strats].dropna().cumsum().apply(np.exp).plot()


# In[10]:


from oandapyV20.endpoints.pricing import PricingStream
import oandapyV20.endpoints.orders as orders
from oandapyV20.contrib.requests import MarketOrderRequest
from oandapyV20.exceptions import V20Error, StreamTerminated
import oandapyV20.endpoints.trades as trades
from oandapyV20.contrib.requests import TrailingStopLossOrderRequest

class MomentumTrader(PricingStream): 
    def __init__(self, momentum, *args, **kwargs): 
        PricingStream.__init__(self, *args, **kwargs)
        self.ticks = 0 
        self.position = 0
        self.df = pd.DataFrame()
        self.momentum = momentum
        self.units = 1000
        self.connected = False
        #self.client = API(access_token=config['oanda']['access_token'])
        self.client = opy.API(access_token=access_token)

    def create_order(self, units):
        #order = orders.OrderCreate(accountID=config['oanda']['account_id'], data=MarketOrderRequest(instrument="EUR_USD", units=units).data)
        order = orders.OrderCreate(accountID=accountID, data=MarketOrderRequest(instrument=instrument, units=units).data)
        response = self.client.request(order)
        print('\t', response)

        #create a stop might need it's own fuction and called under the trading stragety part
        #stp = pd.Series(response, name='stop')
        #stp_id = stp.orderFillTransaction.get('tradeOpened').get('tradeID')
        #print ('\t The tradeID is ', stp_id)
        #stp_vwap = (stp.orderFillTransaction.get('fullVWAP') - 0.00010
        #print ('\t The stop loss value is ', stp_vwap)
        #stp_order = orders.OrderCreate(accountID=accountID, data=StopLossOrderRequest(tradeID=stp_id, price=stp_vwap))
        #response_stp_order = self.client.request(stp_order)

        #create a trailing stop
        #This below is for a trailing stop that works but the distance is key, this should be set for breakouts
        #tsl = pd.Series(response, name='junk')
        #tsl_id = tsl.orderFillTransaction.get('tradeOpened').get('tradeID')
        #print('The trade ID is: ', tsl_id)
        #tsl_order = orders.OrderCreate(accountID=accountID, data=TrailingStopLossOrderRequest(tradeID=tsl_id, distance=5).data)
        #response_tsl = self.client.request(tsl_order)
        #print('\t', response_tsl)


    def on_success(self, data):
        self.ticks += 1
        print("ticks=",self.ticks)
        # print(self.ticks, end=', ')

        # appends the new tick data to the DataFrame object
        self.df = self.df.append(pd.DataFrame([{'time': data['time'],'closeoutAsk':data['closeoutAsk']}],
                                 index=[data["time"]]))
        # transforms the time information to a DatetimeIndex object
        self.df.index = pd.DatetimeIndex(self.df["time"])

        # Convert items back to numeric (Why, OANDA, why are you returning strings?)
        self.df['closeoutAsk'] = pd.to_numeric(self.df["closeoutAsk"],errors='ignore')

        # resamples the data set to a new, homogeneous interval, set this from '5s' to '1m'
        dfr = self.df.resample('60s').last().bfill()

        #print ('\t this is the dfr', dfr)

        # calculates the log returns
        dfr['returns'] = np.log(dfr['closeoutAsk'] / dfr['closeoutAsk'].shift(1))

        #print('/t returns: ', dfr['returns'])

        # derives the positioning according to the momentum strategy
        dfr['position'] = np.sign(dfr['returns'].rolling(self.momentum).mean())

        print("position=",dfr['position'].iloc[-1])

        if dfr['position'].iloc[-1] == 1:
            print("go long")
            if self.position == 0:
                self.create_order(self.units)
            elif self.position == -1:
                self.create_order(self.units * 2)
            self.position = 1

        elif dfr['position'].iloc[-1] == -1:
            print("go short")
            if self.position == 0:
                self.create_order(-self.units)
            elif self.position == 1:
                self.create_order(-self.units * 2)
            self.position = -1

        if self.ticks == 25000:
            print("close out the position")
            if self.position == 1:
                self.create_order(-self.units)
            elif self.position == -1:
                self.create_order(self.units)
            self.disconnect()

    def disconnect(self):
        self.connected=False

    def rates(self, account_id, instruments, **params):
        self.connected = True
        params = params or {}
        ignore_heartbeat = None
        if "ignore_heartbeat" in params:
            ignore_heartbeat = params['ignore_heartbeat']
        while self.connected:
            response = self.client.request(self)
            for tick in response:
                if not self.connected:
                    break
                if not (ignore_heartbeat and tick["type"]=="HEARTBEAT"):
                    print(tick)
                    self.on_success(tick)


# In[11]:


# Set momentum to be the number of previous 60 second intervals to calculate against
# For two hours, momentum = 120. For 15 minutes, set to 15
mt = MomentumTrader(momentum=15,accountID=accountID,params={'instruments': instrument})
print (mt)


# In[12]:


mt.rates(account_id=accountID, instruments=instrument, ignore_heartbeat=True)


# In[ ]:





# In[ ]:





# In[ ]:





# In[ ]:

comments

Working on 2020

Happy New Year readers! Granted it's already 10 days into the new year but I've been having a well deserved break and easing back into work. Now that 2020 is here I hope to amp up more of what I did last year. Read more books, go to the Dojo/exercise more, and make more adventures with the family. I'm not really into the 'new year resolutions' because I like to ease into things, not make abrupt cold turkey changes.

I hope to close out several small lingering projects in 2020 that I started last year and continue writing my weekly Data Science newsletter. It's been a lot of fun writing this short newsletter and packing it full of articles I found interesting over the week. You can sign up for it below this article or here, if interested.

On the work front, I'm going to continue working on my Business Development Rep bot in Python. I started it last year to automate the laborious tasks my BDR and AE were doing. Yes, we all use Salesforce but for me it's faster to just to log into a command line and write a python script to do some extraction and analysis from Salesforce. It feels pretty rewarding to automate those things because it makes everyone's life better and hopefully more lucrative.

As of next week my traveling will begin again. I'm off to Charlotte next week and then to San Francisco and Los Angelos for a week and a half. Somewhere in between that time I have to prioritize my family life and juggle my work life.

That's the trick, prioritize the Big Rocks and juggle the rest.

comments

Python Skills

The thing about coding is that you'll never want to do it unless you have to solve something with it. This is why Python has become an indispensable language for all sorts of Data munging. There's a ton of data generated every day and you can use it for gain (or loss).

Over the past month my Python skills have made a big leap forward. This is because I need to manipulate data inside my organization. I'm putting the finishing touches on a BDR (Business Development Rep) Bot that I built to help my AE (Account Exec). We get a ton of leads everyday and this Python script cuts through the noise to the higher quality leads. Why? So we can focus our limited time and effort on the right leads. I managed to whip this up in about two days. It took that long because I had to teach myself Browser Automation, which is in itself is cool.

I also spent time this week resurrecting the pieces of a Forex Trading Bot. After 12 years of blogging and and wanting to do it, I'm finally putting my Trading Bot into 'production.' I use the word production in quotes because it'll run on Oanda's Practice Account first. I want to see if my initial Momentum Strategy will work or not. If it doesn't, I'll come up with another strategy class and go from there.

The best part? I cobbled it all together with the help of Python and Google. Python for scripting it and Google for figuring out where I got stuck. In reality, I owe all my coding skills to Google search (or DuckDuckGo). Stackoverflow and countless blogs have been invaluable to me.

We live in such amazing times. We can learn so many things online now. It requires YOU to take an interest and have the will to learn.

comments

Neural Market Trends is the online home of Thomas Ott.