Neural Market Trends |||

Auto Generating Blog Posts with RapidMiner

Lately I’ve been fooling around with the new Blot.Im blogging engine and decided to see if I could use RapidMiner and some Javascript to auto generate a blog post with a stock chart. Since blot can parse txt, markdown, and HTML files, I decided to see if I can get this to work with an HTML. [While Markdown has the ability to parse HTML tags, it can’t parse Javascript correctly]1. After some pain, I got it to work with Blot, but you can extend this to any type of blogging system that parses HTML.

I designed a RapidMiner process that downloads the historical stock data using three macros: symbol, start_date, and end_date. These macros are help set the parameters in the Yahoo Stock Data operator but were also used to pass thru to the actual Javascript code.

For the stock chart I used the plot.ly library and passed the %{symbol} macro and %{end_date} to the script. For the sake of syncing up the saved CSV file, I had to careful that it was appended with those macros.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.001">
<context>
<input/>
<output/>
<macros>
<macro>
<key>symbol</key>
<value>VSLR</value>
</macro>
<macro>
<key>end_date</key>
<value>2016-04-14</value>
</macro>
<macro>
<key>start_date</key>
<value>2015-10-14</value>
</macro>
</macros>
</context>
<operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="quantx1:yahoo_historical_data_extractor" compatibility="1.0.006" expanded="true" height="82" name="Yahoo Historical Stock Data" width="90" x="45" y="34">
<parameter key="I agree to abide by Yahoo's Terms &amp; Conditions on financial data usage" value="true"/>
<parameter key="Quick Stock Ticker Data" value="true"/>
<parameter key="Stock Ticker" value="%{symbol}"/>
<parameter key="select_fields" value="CLOSE|OPEN|DAY_LOW|DAY_HIGH"/>
<parameter key="date_format" value="yyyy-MM-dd"/>
<parameter key="date_start" value="%{start_date}"/>
<parameter key="date_end" value="%{end_date}"/>
</operator>
<operator activated="true" class="sort" compatibility="7.0.001" expanded="true" height="82" name="Sort" width="90" x="179" y="34">
<parameter key="attribute_name" value="Date"/>
</operator>
<operator activated="true" class="rename" compatibility="7.0.001" expanded="true" height="82" name="Rename" width="90" x="313" y="34">
<parameter key="old_name" value="%{symbol}_OPEN"/>
<parameter key="new_name" value="open"/>
<list key="rename_additional_attributes">
<parameter key="%{symbol}_DAY_HIGH" value="high"/>
<parameter key="%{symbol}_DAY_LOW" value="low"/>
<parameter key="%{symbol}_CLOSE" value="close"/>
<parameter key="Date" value="dates"/>
</list>
</operator>
<operator activated="true" class="select_attributes" compatibility="7.0.001" expanded="true" height="82" name="Select Attributes" width="90" x="447" y="34">
<parameter key="attribute_filter_type" value="single"/>
<parameter key="attribute" value="dates"/>
<parameter key="invert_selection" value="true"/>
</operator>
<operator activated="true" class="write_csv" compatibility="7.0.001" expanded="true" height="82" name="Write CSV" width="90" x="581" y="34">
<parameter key="csv_file" value="C:\Users\tott_000\Dropbox\Apps\Blot\neuralmarket\public\autocharts\%{symbol}_%{end_date}_data.csv"/>
<parameter key="column_separator" value=","/>
</operator>
<operator activated="true" class="text:create_document" compatibility="7.0.000" expanded="true" height="68" name="Plot.ly" width="90" x="45" y="187">
<parameter key="text" value="&lt;head&gt;&#10;  &lt;!-- Plotly.js --&gt;&#10;  &lt;script src=&quot;https://cdn.plot.ly/plotly-latest.min.js&quot;&gt;&lt;/script&gt;&#10;  &lt;!-- PlotlyFinance.js --&gt;&#10;  &lt;script src=&quot;https://cdn.rawgit.com/etpinard/plotlyjs-finance/master/plotlyjs-finance.js&quot;&gt;&lt;/script&gt;&#10;&#10;&lt;body&gt;&#10;&lt;H1&gt;Today's Chart: %{symbol}&lt;/h1&gt;&#10;&#10;&lt;p&gt;The chart below is created using the Plot.ly JS library. This a daily chart starting from %{start_date} to %{end_date}.&lt;/p&gt;&#10;  &#10;&lt;div id=&quot;myDiv&quot; style=&quot;width: 100%; height: 380px;&quot;&gt;&lt;!-- Plotly chart will be drawn inside this DIV --&gt;&lt;/div&gt;&#10;&lt;script&gt;&#10;   &#9;function makeplot() {&#10;  &#9;Plotly.d3.csv(&quot;http://www.neuralmarkettrends.com/public/autocharts/%{symbol}_%{end_date}_data.csv&quot;, function(data){ processData(data) } );&#10;&#9;};&#10;&#10;&#9;function processData(allRows) {&#10;&#10;  &#9;console.log(allRows);&#10;  &#9;var data_open = [], data_close = [], data_high = [], data_low = [], dates = [];&#10;  &#9;for (var i=0; i&lt;allRows.length; i++) {&#10;   &#9; row = allRows[i];&#10;     data_close.push(parseFloat(row['close']));&#10;     data_high.push(parseFloat(row['high']));&#10;     data_low.push(parseFloat(row['low']));  &#10;     data_open.push(parseFloat(row['open']));&#10; &#9; }&#10;&#9;  makePlotly( data_open, data_close, data_high, data_low ); &#10;&#9;}&#10;&#10;&#10;&#9;function makePlotly( data_open, data_close, data_high, data_low ){&#10;&#9;    var data_dates = getAllDays('%{start_date}', '%{end_date}');&#10;    &#10;&#9;    var fig = PlotlyFinance.createCandlestick({&#10;&#9;    open: data_open,&#10;&#9;    high: data_high,&#10; &#9;    low: data_low,&#10;&#9;    close: data_close,&#10;&#9;    dates: data_dates&#10;&#9;   });&#10;&#10;&#9;fig.layout.title = 'Daily Stock Chart';&#10;&#10;&#9;    fig.layout.annotations = [&#10;&#9;                 {&#10;&#9;                     text: &quot;%{symbol} Stock&quot;,&#10;&#9;                     x: '-0.05',&#10;&#9;                     y: 0.5,&#10;&#9;                     xref: 'paper',&#10;&#9;                     yref: 'paper',&#10;     &#9;                font:{&#10;          &#9;                 size: 18&#10;               &#9;           },&#10;                    &#9; showarrow: false,&#10;&#9;                     xanchor: 'right',&#10;&#9;                     textangle: 270&#10;     &#9;            }&#10;          &#9;      ];&#10;&#10;&#9;Plotly.newPlot('myDiv', fig.data, fig.layout);&#10;&#9;};&#10;&#10;// Utility Function to generate all days&#10;function getAllDays(start, end) {&#10;    var s = new Date(start);&#10;    var e = new Date(end);&#10;    var a = [];&#10;    &#10;    while(s &lt; e) {&#10;        a.push(s);&#10;        s = new Date(s.setDate(&#10;            s.getDate() + 1&#10;        ))&#10;    }&#10;&#10;    return a;&#10;};&#10;&#10;makeplot();&#10;&lt;/script&gt;&#10;&lt;/body&gt;&#10;"/>
</operator>
<operator activated="true" class="text:write_document" compatibility="7.0.000" expanded="true" height="82" name="Convert format" width="90" x="179" y="187"/>
<operator activated="true" class="write_file" compatibility="7.0.001" expanded="true" height="68" name="Write to HTML format (3)" width="90" x="313" y="187">
<parameter key="filename" value="C:\Users\tott_000\Dropbox\Apps\Blot\neuralmarket\2016\%{end_date}-%{symbol}-chart.html"/>
</operator>
<connect from_op="Yahoo Historical Stock Data" from_port="example set" to_op="Sort" to_port="example set input"/>
<connect from_op="Sort" from_port="example set output" to_op="Rename" to_port="example set input"/>
<connect from_op="Rename" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
<connect from_op="Select Attributes" from_port="example set output" to_op="Write CSV" to_port="input"/>
<connect from_op="Plot.ly" from_port="output" to_op="Convert format" to_port="document"/>
<connect from_op="Convert format" from_port="file" to_op="Write to HTML format (3)" to_port="file"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
</process>
  </operator>
</process>

While this works really great, it does have one snag. If you want to display multiple stockcharts, you’d have to figure out a way to update the location of the chart relative to where your blog post is.

This is important if you show more than one post per page. When I generated two posts with two different stockcharts, the HTML made them relative to the top of the page. In other words, the one chart was on top of the other one. I’m sure this is an easy fix but something I’m not going to bother with for this tutorial!


  1. Nope, got it to work in Markdown with the help of https://twitter.com/lllIIlIlIl

Up next Today's Chart: MCD The chart of $MCD is generated using the Plot.ly JS library and RapidMiner Studio. This is a daily chart from 2015-10-18 to 2016-04-18. For further Today's Chart: FBIO The chart of $FBIO is generated using the Plot.ly JS library and RapidMiner Studio. This is a daily chart from 2015-10-18 to 2016-04-18. For further
Latest posts Democratising Machine learning with H2O — Towards Data Science Getting started with Python datatable | Kaggle Phone Addiction Version 12 Launches Today! Machine Learning Making Pesto Tastier 5 Dangerous Things You Should Let Your Kids Do The Pyschology of Writing TensorFlow and High Level APIs Driving Marketing Performance with H2O Driverless AI Machine Learning and Data Munging in H2O Driverless AI with datatable Making AI Happen Without Getting Fired Latest Musings from a Traveling Sales Engineer The Night before H2O World 2019 Why Forex Trading is Frustrating Functional Programming in Python Automatic Feature Engineering with Driverless AI Ray Dalio's Pure Alpha Fund What's new in Driverless AI? Latest Writings Elsewhere - December 2018 House Buying Guide for Millennials Changing Pinboard Tags with Python Automate Feed Extraction and Posting it to Twitter Flux: A Machine Learning Framework for Julia Getting Started in Data Science Part 2 Makers vs Takers How Passive Investing Saved My Life Startups and Open Source The Process of Writing H2O AI World 2018 in London Ray Dalio's Pure Alpha Fund Isolation Forests in H2O.ai