Tag Twitter

Posts: 11

Automate Feed Extraction and Posting it to Twitter

I wrote a small Python script to automate feed extraction and posting it to Twitter. The feed? My feed burner feed for this site. The script is super simple and uses the Twython, Pandas, and Feedparser libraries. The rest, I believe, comes stock with your Python 3 installation.

How it works

It's really a simple process. First it takes a feed, then parses it, does a few things to it, saves it to a CSV file, and text file. Then it opens the text file (not elegant , I know) and randomly selects a row (blog post), and posts it to Twitter. I use this script to compile all my blog posts into a handy link formatted list that I use in my Newsletter.

There are a few things you need to have to run the Twitter part of the script. You'll need to create an API key from Twitter and use your Consumer and Secret tokens to run the Twitter posting.

A lot of this script came from my original one here. I just added in some time filtering procedures. For example:

today = DT.date.today()
week_ago = today - DT.timedelta(days=7)
month_ago = today - DT.timedelta(days=60)

Sets up the time for today, a week ago and a month ago from today's date. I use them to set my time filter in section 4 below and get a list of blog posts posted in the last 7 days or 60 days. You may change these values to your heart's content. The script is open source and free to use anyway you want.

Python Code

# coding: utf-8

# In[1]:

#!/usr/bin/env python
import sys
import os
import random

from twython import Twython, TwythonError

from keys import dict
import datetime as DT
import feedparser

import pandas as pd
import csv

# In[2]:

#Set Time Deltas for tweets

today = DT.date.today()
week_ago = today - DT.timedelta(days=7)
month_ago = today - DT.timedelta(days=60)

t = today.timetuple()
w = week_ago.timetuple()
m = month_ago.timetuple()

# In[3]:

#Parse the Feed
d = feedparser.parse('https://www.neuralmarkettrends.com/feed/')

# In[4]:

#Create List of Feed items and iterate over them. 
output_posts = []
for pub_date in d.entries:
    date = pub_date.published_parsed
    #I need to automate this part below
    if date >= m and date <= t:
        print (pub_date.title   ' : '   pub_date.link)
        tmp = pub_date.title,pub_date.link

# In[5]:


#Create Dataframe for easy saving later
df = pd.DataFrame(output_posts, columns = ["Title", "RawLink"])

# In[6]:

date_f = str(DT.date.today())
#f = open (date_f   '-posts.md', 'w')
f = open ('60daytweet.txt', 'w')
for t in output_posts:
    line = ' : '.join(str(x) for x in t)
    f.write(line   '\n')

# In[7]:

#Create Preformated link to export to CSV
df['Hyperlink'] = '<a href="' df['RawLink'] '">' df['Title'] '</a>'

# In[8]:

#Write to CSV for Newsletter
df.to_csv("formated_link_list.csv", quoting=csv.QUOTE_NONE)

# In[9]:

#Initialize automatic Twitter posting of random blog article

#Add your Twitter API keys by replacing 'dict['ckey|csecret|atoken|asecret'] with the applicable values like so 'XXXXXX'

CONSUMER_KEY = dict['ckey'] 
CONSUMER_SECRET = dict['csecret']
ACCESS_KEY = dict['atoken']
ACCESS_SECRET = dict['asecret']

f = open('60daytweet.txt', 'r')

lines = f.readlines()

post = random.choice(lines)


except TwythonError as e:
    print (e)

Note: This code was first sent out as part of my monthly newsletter. You can sign up for it here.


The Fallacy of Twitter Bots

I'm going to be the first to admit that I use Python to send out Tweets to my followers. I have a few scripts that parse RSS feeds and do retweets on an hourly basis. They work fine but they do get 'gamed' occasionally. That's the problem with automation, isn't it? Getting gamed can cause all kinds of havoc for your brand and reputation, so you have to be careful.

Has this happened to me? Not really, but there has been a few embarrassing retweets and silly parsed advertisements in lieu of good articles.

Why bother with Twitter automation in the first place? Simple, everyone wants to be an 'influencer', myself included. Yet using automated methods to gain 'eyeballs' comes with a price. You end up sacrificing quality for quantity. You end up diluting your brand and losing the signal. In the end you get nothing but noise!

Signal vs Noise

At one time I tested/used @randal_olson's TwitterFollowBot to increase my follower count. It worked well and I started growing my followers in large clips. The script is pretty simple in logic, it follows people based on a certain hashtag (or followers of a Twitter handle) that you supply and does in about 100 people per run.

The goal here is to get a 'follow back' from the people you just followed, then auto mute them. If, after a week or so, they don't follow you back you run another routine that 'unfollows' them and puts them on a black list not to 'autofollow' them again.

You run this script every few hours for a week and MY GAWD, does your following list explode! The noise becomes unbearable, even after muting them. You end up with cranks, conspiracy theorists, crypto-currency shills, and bots (most liked Russian bots). Yes, you do get a lot of follow backs but the quality signal of people you should really follow and interact with gets completely lost!

I stopped that experiment a few months ago and started unfollowing the noise. My following count is now below 1,000 but I feel that's too much. I want to get that number to about 500. Of course, this resulted in my follower count dropping too. There's a lot of Twitter users that also run 'you unfollow me so I unfollow you' scripts too. LOL.

Possible solutions

Just stop it. Stop all the Retweeting, TwitterBot following, and parsing. Instead do one or more of the following:

  1. Create a curated list of great links that you filter through. I know that @maoxian has done this over the years and it's invaluable because he puts the time and effort in to filtering out the noise.
  2. Write a Python script to parse RSS feeds but write the links to a file so you can review later and tweet accordingly (more signal, less noise)
  3. Write a Python script to find 'true' influencers on Twitter and interact with them personally. Perhaps create a ranking system
  4. Something else that I'll remember after I post this article

I guess lesson here is that we can't automate the human touch. You can do a lot of the heavy lifting but in the end, it's us that bring meaning and value to everything we do.


Word2Vec Example Process in RapidMiner

This is an example process of how to use Word2Vec in RapidMiner with the Search Twitter operator. For more information check out this post on the community.

I'll be going over this in a bit more detail at my next live stream here.


        <?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
          <operator activated="true" class="process" compatibility="8.1.001" expanded="true" name="Process">
            <process expanded="true">
              <operator activated="true" class="social_media:search_twitter" compatibility="8.1.000" expanded="true" height="68" name="Search Twitter" width="90" x="45" y="34">
                <parameter key="connection" value="Twitter - Studio Connection"/>
                <parameter key="query" value="rapidminer"/>
                <parameter key="locale" value="en"/>
              <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes" width="90" x="45" y="136">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="Text"/>
                <parameter key="include_special_attributes" value="true"/>
              <operator activated="true" class="nominal_to_text" compatibility="8.1.001" expanded="true" height="82" name="Nominal to Text" width="90" x="45" y="238"/>
              <operator activated="true" class="text:data_to_documents" compatibility="8.1.000" expanded="true" height="68" name="Data to Documents" width="90" x="246" y="34">
                <list key="specify_weights"/>
              <operator activated="true" class="loop_collection" compatibility="8.1.001" expanded="true" height="82" name="Loop Collection" width="90" x="246" y="136">
                <process expanded="true">
                  <operator activated="true" class="text:transform_cases" compatibility="8.1.000" expanded="true" height="68" name="Transform Cases" width="90" x="112" y="34"/>
                  <operator activated="true" class="text:tokenize" compatibility="8.1.000" expanded="true" height="68" name="Tokenize" width="90" x="581" y="34"/>
                  <connect from_port="single" to_op="Transform Cases" to_port="document"/>
                  <connect from_op="Transform Cases" from_port="document" to_op="Tokenize" to_port="document"/>
                  <connect from_op="Tokenize" from_port="document" to_port="output 1"/>
                  <portSpacing port="source_single" spacing="0"/>
                  <portSpacing port="sink_output 1" spacing="0"/>
                  <portSpacing port="sink_output 2" spacing="0"/>
              <operator activated="true" class="word2vec:Word2Vec_Learner" compatibility="1.0.000" expanded="true" height="68" name="Word2Vec " width="90" x="447" y="34"/>
              <operator activated="true" class="word2vec:Get_Vocabulary" compatibility="1.0.000" expanded="true" height="82" name="Extract Vocabulary" width="90" x="581" y="34">
                <parameter key="Get Full Vocabulary" value="true"/>
                <parameter key="Take Random Words" value="false"/>
                <parameter key="Number of Words to Pull" value="10"/>
              <connect from_op="Search Twitter" from_port="output" to_op="Select Attributes" to_port="example set input"/>
              <connect from_op="Select Attributes" from_port="example set output" to_op="Nominal to Text" to_port="example set input"/>
              <connect from_op="Nominal to Text" from_port="example set output" to_op="Data to Documents" to_port="example set"/>
              <connect from_op="Data to Documents" from_port="documents" to_op="Loop Collection" to_port="collection"/>
              <connect from_op="Loop Collection" from_port="output 1" to_op="Word2Vec " to_port="doc"/>
              <connect from_op="Word2Vec " from_port="mod" to_op="Extract Vocabulary" to_port="mod"/>
              <connect from_op="Extract Vocabulary" from_port="exa" to_port="result 1"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_result 1" spacing="0"/>
              <portSpacing port="sink_result 2" spacing="0"/>


Enable Twitter Cards and Open Graph in Blot.Im

I struggled for days on how to correctly enable Twitter Cards and Open Graph (aka Facebook) in my Blot.IM templates! Days!

The problem is now solved and I will share it here with fellow Blotsters. I have to give a big shouout to David from Blot, I couldn't have solved it without his help!

Why Twitter Cards and Open Graph?

Let's be honest here, not many people create a blog or some sort of content just for it to sit hidden on the Internet. We all crave 'eyeballs' and interaction from others for our awesome content! For that we have to share our missives using mostly Twitter and Facebook. Wordpress and other CMS's make this really easy out of the box via their fancy templates but Blot is a bit different.

Granted it's an emerging awesome CMS and has some great advantages over the Wordpress bloat (see more here), it does require you to roll up your sleeves and do some simple hacking from time to time.

Twitter Cards

I started out by wanting to implement Twitter Cards. If I could that working, then I'd move on to Facebook. As a baseline, I went to the Twitter Card Validator and checked a random blog post. It failed miserably!!! I really wanted to embed an image from the post itself into the Twitter card so I don't get that lame grey icon. I wanted the twitter card to look like below for the this post.

Twitter Card code in your Head file

The neat thing about Blot.Im is that you can hack the HTML files of your theme really easily. You can do it in the native editor Blot provides OR you can do it via importing the templates locally. I work locally because I like working in Sublime Text better than going to the website.

I first opened the head.html file and inspected added a new section called <---Twitter Code--->. Then after much trial and error, and with David's help, figured out the right way hack in the mustache template code.

This works great as long as you have either one image in the body of the blog post. Next step was hacking for Facebook!

Facebook Open Graph

Building on what I learned from above, merging in the Open Graph calls were quite easy.

There was one snag though, you need to get a FB API key to get this work. In the image above you can see where it goes. That's it. Pretty simple if you know what you are doing!


Best Adsense month so far

Last month I made $6.51 from Adsense revenue, the best month so far since I started this experiment. Although I didn’t hit the magic “1 roll of Portra 400” mark, it came pretty close.


I credit a lot of the new Adsense revenue to switching to a WordPress theme and using the Adsense plugin by Google. Everything appears to be better optimized, but I’m sure I could do more.

Python Script Experiments

I also started experimenting with some modified Python scripts to automate some of my Twitter tasks. My automation bot R2D2 spends time each morning scanning popular #ai and #machinelearning posts and then retweets them.

Since I’ve been doing that, I’ve noticed a deluge of Twitter users putting me on a list. I suspect that’s some sort of Bot scanning retweets and then auto populating me on a list. I will monitor this as a I go along.

I’ve also noticed a bump in new followers but also a strange unfollowing within 24 hours. I think there is some sort of automated script running that autofollows me in the hope that I’ll follow them back and then it unfollows me. I’ve noticed the same handful of Tweeple follow me and then follow me again. So they must be unfollowing me between the two follows.



Neural Market Trends is the online home of Thomas Ott.