Tag Automation

Posts: 7

Functional Programming in Python

I'm spending time trying to understand the differences between writing classes and functions in Python. Which one is better and why? From what I'm gathering, a lot of people are tired of writing classes in general. Classes are used in Object Oriented Programming (OOP) and some python coders hate it because it's writing too many lines of code when only a few really matter. So programmers like functional programming (FP) in python instead.

To that end, I've been watching videos of both. OOP and FP videos on the Internet and started writing notes on them. Below is a great but also very deep video on functional progamming in python by Daniel Kirsch from PyData 2016. It's a great video and his presentation is about 30 minutes with a great Q&A session.

https://youtu.be/r2eZ7lhqzNE

Functional Programming in Python

My notes from the above video are above are below:

  • First Class Functions
  • Higher Order Functions
  • Purity
  • Immutability (not going to talk about it)
  • Composition
  • Partial Application & Currying
  • Purity, a function without 'side effects'
  • First Class Functions, simply means that functions are like everybody else
  • Can define with 'def' or lambda
  • Can use the name of functions as variables and do higher-order programming
  • Decorators "… provide a simple syntax for calling higher-order functions. By definition, a decorator is a function that takes another function and extends the behavior of the latter function without explicitly modifying it."
  • Partial function applications - "The primary tool supplied by the Functools module is the class partial, which can be used to “wrap” a callable object with default arguments. Partial objects are similar to function objects with slight differences. Partial function application makes it easier to write and maintain the code."
  • Partial functions are very powerful
  • "Currying transforms a function that takes multiple arguments in such a way that it can be called as a chain of functions. Each with a single argument (Partial Application)." via Wikipedia
  • The important concept for Currying is closures, aka lexical scoping
  • Remembers the variables in the scope where it was defined
  • List comprehensions vs functional equivalents
  • Map function vs list comprehension
  • Filter function vs list comprehension
  • Reduce vs list comprehension
  • Why not write out the loop instead? Using Map/Filter/Reduce is cleaner
  • Function composition: i.e. run a filter and then map: map(f, filter(p, seq))
  • 'Import functools' is very useful
  • Main takeaways: Function Programming is possible in Python (to a degree)
  • Main takeaways: Small composable function are good
  • Main takeaways: FP == Build General Tools and Compose them
  • Python is missing: more list functions
  • Python is missing: Nicer lambda syntax
  • Python is missing: Automatic currying, composition syntax
  • Python is missing: ADTS (Sum Types)
  • Python is missing: Pattern Matching
  • Some remedies for list functions
  • Links provide in video @ 26:00
  • Suggest learning Haskell as a gateway to functional programming.

comments

Changing Pinboard Tags with Python

Welcome to another automation post! This is a super simple Python script for changing misspelled or wrong tags in your Pinboard account. I started using Pinboard again because it helps me save all these great articles I read on the Interwebz, so I can paraphrase and regurgitate them back to you. Ha!

I need to clean out the Pinboard tags every so often because I hooked it up to Twitter. It works well for me because it saves all my retweets, favs and posts, but there's a lot of noise. Sometimes I end up with tags like "DataScience" and "DataScientists" when I really want "DataScience." I did some searching around and found the Pinboard Python library. Changing Pinboard tags with Python is EASY!

What you do need to do is install the Python package for Pinboard and get an API key from your Settings page. Then it's as simple as doing this:

Python Code

import pinboard

pb = pinboard.Pinboard('INSERT_YOUR_API_KEY_HERE')

old_tag = 'DataMining'
new_tag = 'DataScience'

pb.tags.rename(old=old_tag, new=new_tag)

You can, of course modify this script to pass command line arguments to it and just do something like this:

import pinboard
import sys

passcode = str(input('Enter your Pinboard API key here: '))

pb = pinboard.Pinboard(passcode)

old_tag = str(input('Enter the old tag: '))
new_tag = str(input('Enter the new tag: '))

pb.tags.rename(old=old_tag, new=new_tag)

print ('Converted: ' + old_tag+ ' to: ' + new_tag) 

Once again, the second script is all open source and free for you to use/modify as you see fit.

Note: I just regurgitated the original script (first one) and then riffed on it for the second one. The Author of Pinboard provided a sample in the documentation. Check that out too!

comments

Automate Feed Extraction and Posting it to Twitter

I wrote a small Python script to automate feed extraction and posting it to Twitter. The feed? My feed burner feed for this site. The script is super simple and uses the Twython, Pandas, and Feedparser libraries. The rest, I believe, comes stock with your Python 3 installation.

How it works

It's really a simple process. First it takes a feed, then parses it, does a few things to it, saves it to a CSV file, and text file. Then it opens the text file (not elegant , I know) and randomly selects a row (blog post), and posts it to Twitter. I use this script to compile all my blog posts into a handy link formatted list that I use in my Newsletter.

There are a few things you need to have to run the Twitter part of the script. You'll need to create an API key from Twitter and use your Consumer and Secret tokens to run the Twitter posting.

A lot of this script came from my original one here. I just added in some time filtering procedures. For example:

today = DT.date.today()
week_ago = today - DT.timedelta(days=7)
month_ago = today - DT.timedelta(days=60)

Sets up the time for today, a week ago and a month ago from today's date. I use them to set my time filter in section 4 below and get a list of blog posts posted in the last 7 days or 60 days. You may change these values to your heart's content. The script is open source and free to use anyway you want.

Python Code

# coding: utf-8

# In[1]:


#!/usr/bin/env python
import sys
import os
import random

from twython import Twython, TwythonError

from keys import dict
import datetime as DT
import feedparser

import pandas as pd
import csv


# In[2]:


#Set Time Deltas for tweets

today = DT.date.today()
week_ago = today - DT.timedelta(days=7)
month_ago = today - DT.timedelta(days=60)

t = today.timetuple()
w = week_ago.timetuple()
m = month_ago.timetuple()


# In[3]:


#Parse the Feed
d = feedparser.parse('https://www.neuralmarkettrends.com/feed/')


# In[4]:


#Create List of Feed items and iterate over them. 
output_posts = []
for pub_date in d.entries:
    date = pub_date.published_parsed
    #I need to automate this part below
    if date >= m and date <= t:
        print (pub_date.title   ' : '   pub_date.link)
        tmp = pub_date.title,pub_date.link
        output_posts.append(tmp)


# In[5]:


#print(output_posts)

#Create Dataframe for easy saving later
df = pd.DataFrame(output_posts, columns = ["Title", "RawLink"])


# In[6]:


date_f = str(DT.date.today())
#f = open (date_f   '-posts.md', 'w')
f = open ('60daytweet.txt', 'w')
for t in output_posts:
    line = ' : '.join(str(x) for x in t)
    f.write(line   '\n')
f.close()


# In[7]:


#Create Preformated link to export to CSV
df['Hyperlink'] = '<a href="' df['RawLink'] '">' df['Title'] '</a>'


# In[8]:


#Write to CSV for Newsletter
df.to_csv("formated_link_list.csv", quoting=csv.QUOTE_NONE)


# In[9]:


#Initialize automatic Twitter posting of random blog article

#Add your Twitter API keys by replacing 'dict['ckey|csecret|atoken|asecret'] with the applicable values like so 'XXXXXX'

CONSUMER_KEY = dict['ckey'] 
CONSUMER_SECRET = dict['csecret']
ACCESS_KEY = dict['atoken']
ACCESS_SECRET = dict['asecret']

f = open('60daytweet.txt', 'r')

lines = f.readlines()
f.close()

post = random.choice(lines)

api = Twython(CONSUMER_KEY,CONSUMER_SECRET,ACCESS_KEY,ACCESS_SECRET)

try: 
    api.update_status(status=post)
except TwythonError as e:
    print (e)

Note: This code was first sent out as part of my monthly newsletter. You can sign up for it here.

comments

Ray Dalio's Pure Alpha Fund

Ray Dalio's Pure Alpha Fund returned 14.6% for 2018. That's an amazing feat considering the majority of hedge funds averaged a loss of 6.7%. How does he do it? Simple. He always asks, "how do I know I'm right?"

The Westport, Connecticut-based firm is the world’s biggest hedge fund with about $160 billion in assets. The gains for its Pure Alpha Strategy came as other fund managers were whipsawed by volatile markets, resulting in the industry posting one of its worst years ever.

Via Bloomberg

How do I know I'm right?

I can't begin to guess how his Alpha Funds works internally, how complex it must be. If his book is any indication (a great summary here), then I can guess it's well thought out with one simple premise, "how do I know I'm right?"

Think about it, 1,000's of Hedge Funds are operating on similar strategies all the time. They're either macro, long/short, arbitrage, whatever... What does he do that's so different than all the rest? The nugget of truth is in his in Ray Dalio's questioning himself all the time with "how do I know I'm right?"

If you believe the market is going to crash when it's at an all-time high, how do check yourself? How, in your infinite wisdom, do you test and confirm that you are correct and the 1,000's of fund manager and quants have it wrong?

History Rhymes

Another great aspect of Ray Dalio's strategy is that he looks in history where a current similar situation has occurred. According to him, he goes back about 500 years in economic/political/cultural history and looks if something similar (debt crisis, tulip mania, etc) has happened and what the outcome was. Then he adjusts his strategy accordingly.

One great example was when the US abandoned the gold standard in 1971. His initial thought was that the market was going to crash but was shocked to learn that the markets opened higher. Later he learned that history is littered with a few examples where this has happened before. The inflationary result of abandoning the gold standard lifted asset classes higher.

While history doesn't necessarily repeat itself, it tends to rhyme a lot. Perfect current example: Bitcoin. Bitcoin is a lot like the Tulip Mania (and other manias for that fact). While I'm bullish on Blockchain technology, I'm not sure if Bitcoin will survive to become a sorely needed digital currency.

No matter what industry you come from, you should always ask yourself if you're right and how can you confirm it. Once you've identified the known-knowns (the market already has), the known-unknowns (maybe the market has), and the unknown-unknowns (no one has, but history might provide a guide), then you can apply strategies and processes to win.

Now, ask yourself: How do I know I'm right?

comments

The Fallacy of Twitter Bots

I'm going to be the first to admit that I use Python to send out Tweets to my followers. I have a few scripts that parse RSS feeds and do retweets on an hourly basis. They work fine but they do get 'gamed' occasionally. That's the problem with automation, isn't it? Getting gamed can cause all kinds of havoc for your brand and reputation, so you have to be careful.

Has this happened to me? Not really, but there has been a few embarrassing retweets and silly parsed advertisements in lieu of good articles.

Why bother with Twitter automation in the first place? Simple, everyone wants to be an 'influencer', myself included. Yet using automated methods to gain 'eyeballs' comes with a price. You end up sacrificing quality for quantity. You end up diluting your brand and losing the signal. In the end you get nothing but noise!

Signal vs Noise

At one time I tested/used @randal_olson's TwitterFollowBot to increase my follower count. It worked well and I started growing my followers in large clips. The script is pretty simple in logic, it follows people based on a certain hashtag (or followers of a Twitter handle) that you supply and does in about 100 people per run.

The goal here is to get a 'follow back' from the people you just followed, then auto mute them. If, after a week or so, they don't follow you back you run another routine that 'unfollows' them and puts them on a black list not to 'autofollow' them again.

You run this script every few hours for a week and MY GAWD, does your following list explode! The noise becomes unbearable, even after muting them. You end up with cranks, conspiracy theorists, crypto-currency shills, and bots (most liked Russian bots). Yes, you do get a lot of follow backs but the quality signal of people you should really follow and interact with gets completely lost!

I stopped that experiment a few months ago and started unfollowing the noise. My following count is now below 1,000 but I feel that's too much. I want to get that number to about 500. Of course, this resulted in my follower count dropping too. There's a lot of Twitter users that also run 'you unfollow me so I unfollow you' scripts too. LOL.

Possible solutions

Just stop it. Stop all the Retweeting, TwitterBot following, and parsing. Instead do one or more of the following:

  1. Create a curated list of great links that you filter through. I know that @maoxian has done this over the years and it's invaluable because he puts the time and effort in to filtering out the noise.
  2. Write a Python script to parse RSS feeds but write the links to a file so you can review later and tweet accordingly (more signal, less noise)
  3. Write a Python script to find 'true' influencers on Twitter and interact with them personally. Perhaps create a ranking system
  4. Something else that I'll remember after I post this article

I guess lesson here is that we can't automate the human touch. You can do a lot of the heavy lifting but in the end, it's us that bring meaning and value to everything we do.

comments

Neural Market Trends is the online home of Thomas Ott.