April 18, 2016

Auto Generating Blog Posts with RapidMiner

Lately I’ve been fooling around with the new Blot.Im blogging engine and decided to see if I could use RapidMiner and some Javascript to auto generate a blog post with a stock chart. Since blot can parse txt, markdown, and HTML files, I decided to see if I can get this to work with an HTML. [While Markdown has the ability to parse HTML tags, it can’t parse Javascript correctly]1. After some pain, I got it to work with Blot, but you can extend this to any type of blogging system that parses HTML.

I designed a RapidMiner process that downloads the historical stock data using three macros: symbol, start_date, and end_date. These macros are help set the parameters in the Yahoo Stock Data operator but were also used to pass thru to the actual Javascript code.

For the stock chart I used the plot.ly library and passed the %{symbol} macro and %{end_date} to the script. For the sake of syncing up the saved CSV file, I had to careful that it was appended with those macros.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.001">
<context>
<input/>
<output/>
<macros>
<macro>
<key>symbol</key>
<value>VSLR</value>
</macro>
<macro>
<key>end_date</key>
<value>2016-04-14</value>
</macro>
<macro>
<key>start_date</key>
<value>2015-10-14</value>
</macro>
</macros>
</context>
<operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="quantx1:yahoo_historical_data_extractor" compatibility="1.0.006" expanded="true" height="82" name="Yahoo Historical Stock Data" width="90" x="45" y="34">
<parameter key="I agree to abide by Yahoo's Terms &amp; Conditions on financial data usage" value="true"/>
<parameter key="Quick Stock Ticker Data" value="true"/>
<parameter key="Stock Ticker" value="%{symbol}"/>
<parameter key="select_fields" value="CLOSE|OPEN|DAY_LOW|DAY_HIGH"/>
<parameter key="date_format" value="yyyy-MM-dd"/>
<parameter key="date_start" value="%{start_date}"/>
<parameter key="date_end" value="%{end_date}"/>
</operator>
<operator activated="true" class="sort" compatibility="7.0.001" expanded="true" height="82" name="Sort" width="90" x="179" y="34">
<parameter key="attribute_name" value="Date"/>
</operator>
<operator activated="true" class="rename" compatibility="7.0.001" expanded="true" height="82" name="Rename" width="90" x="313" y="34">
<parameter key="old_name" value="%{symbol}_OPEN"/>
<parameter key="new_name" value="open"/>
<list key="rename_additional_attributes">
<parameter key="%{symbol}_DAY_HIGH" value="high"/>
<parameter key="%{symbol}_DAY_LOW" value="low"/>
<parameter key="%{symbol}_CLOSE" value="close"/>
<parameter key="Date" value="dates"/>
</list>
</operator>
<operator activated="true" class="select_attributes" compatibility="7.0.001" expanded="true" height="82" name="Select Attributes" width="90" x="447" y="34">
<parameter key="attribute_filter_type" value="single"/>
<parameter key="attribute" value="dates"/>
<parameter key="invert_selection" value="true"/>
</operator>
<operator activated="true" class="write_csv" compatibility="7.0.001" expanded="true" height="82" name="Write CSV" width="90" x="581" y="34">
<parameter key="csv_file" value="C:\Users\tott_000\Dropbox\Apps\Blot\neuralmarket\public\autocharts\%{symbol}_%{end_date}_data.csv"/>
<parameter key="column_separator" value=","/>
</operator>
<operator activated="true" class="text:create_document" compatibility="7.0.000" expanded="true" height="68" name="Plot.ly" width="90" x="45" y="187">
<parameter key="text" value="&lt;head&gt;&#10;  &lt;!-- Plotly.js --&gt;&#10;  &lt;script src=&quot;https://cdn.plot.ly/plotly-latest.min.js&quot;&gt;&lt;/script&gt;&#10;  &lt;!-- PlotlyFinance.js --&gt;&#10;  &lt;script src=&quot;https://cdn.rawgit.com/etpinard/plotlyjs-finance/master/plotlyjs-finance.js&quot;&gt;&lt;/script&gt;&#10;&#10;&lt;body&gt;&#10;&lt;H1&gt;Today's Chart: %{symbol}&lt;/h1&gt;&#10;&#10;&lt;p&gt;The chart below is created using the Plot.ly JS library. This a daily chart starting from %{start_date} to %{end_date}.&lt;/p&gt;&#10;  &#10;&lt;div id=&quot;myDiv&quot; style=&quot;width: 100%; height: 380px;&quot;&gt;&lt;!-- Plotly chart will be drawn inside this DIV --&gt;&lt;/div&gt;&#10;&lt;script&gt;&#10;   &#9;function makeplot() {&#10;  &#9;Plotly.d3.csv(&quot;http://www.neuralmarkettrends.com/public/autocharts/%{symbol}_%{end_date}_data.csv&quot;, function(data){ processData(data) } );&#10;&#9;};&#10;&#10;&#9;function processData(allRows) {&#10;&#10;  &#9;console.log(allRows);&#10;  &#9;var data_open = [], data_close = [], data_high = [], data_low = [], dates = [];&#10;  &#9;for (var i=0; i&lt;allRows.length; i++) {&#10;   &#9; row = allRows[i];&#10;     data_close.push(parseFloat(row['close']));&#10;     data_high.push(parseFloat(row['high']));&#10;     data_low.push(parseFloat(row['low']));  &#10;     data_open.push(parseFloat(row['open']));&#10; &#9; }&#10;&#9;  makePlotly( data_open, data_close, data_high, data_low ); &#10;&#9;}&#10;&#10;&#10;&#9;function makePlotly( data_open, data_close, data_high, data_low ){&#10;&#9;    var data_dates = getAllDays('%{start_date}', '%{end_date}');&#10;    &#10;&#9;    var fig = PlotlyFinance.createCandlestick({&#10;&#9;    open: data_open,&#10;&#9;    high: data_high,&#10; &#9;    low: data_low,&#10;&#9;    close: data_close,&#10;&#9;    dates: data_dates&#10;&#9;   });&#10;&#10;&#9;fig.layout.title = 'Daily Stock Chart';&#10;&#10;&#9;    fig.layout.annotations = [&#10;&#9;                 {&#10;&#9;                     text: &quot;%{symbol} Stock&quot;,&#10;&#9;                     x: '-0.05',&#10;&#9;                     y: 0.5,&#10;&#9;                     xref: 'paper',&#10;&#9;                     yref: 'paper',&#10;     &#9;                font:{&#10;          &#9;                 size: 18&#10;               &#9;           },&#10;                    &#9; showarrow: false,&#10;&#9;                     xanchor: 'right',&#10;&#9;                     textangle: 270&#10;     &#9;            }&#10;          &#9;      ];&#10;&#10;&#9;Plotly.newPlot('myDiv', fig.data, fig.layout);&#10;&#9;};&#10;&#10;// Utility Function to generate all days&#10;function getAllDays(start, end) {&#10;    var s = new Date(start);&#10;    var e = new Date(end);&#10;    var a = [];&#10;    &#10;    while(s &lt; e) {&#10;        a.push(s);&#10;        s = new Date(s.setDate(&#10;            s.getDate() + 1&#10;        ))&#10;    }&#10;&#10;    return a;&#10;};&#10;&#10;makeplot();&#10;&lt;/script&gt;&#10;&lt;/body&gt;&#10;"/>
</operator>
<operator activated="true" class="text:write_document" compatibility="7.0.000" expanded="true" height="82" name="Convert format" width="90" x="179" y="187"/>
<operator activated="true" class="write_file" compatibility="7.0.001" expanded="true" height="68" name="Write to HTML format (3)" width="90" x="313" y="187">
<parameter key="filename" value="C:\Users\tott_000\Dropbox\Apps\Blot\neuralmarket\2016\%{end_date}-%{symbol}-chart.html"/>
</operator>
<connect from_op="Yahoo Historical Stock Data" from_port="example set" to_op="Sort" to_port="example set input"/>
<connect from_op="Sort" from_port="example set output" to_op="Rename" to_port="example set input"/>
<connect from_op="Rename" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
<connect from_op="Select Attributes" from_port="example set output" to_op="Write CSV" to_port="input"/>
<connect from_op="Plot.ly" from_port="output" to_op="Convert format" to_port="document"/>
<connect from_op="Convert format" from_port="file" to_op="Write to HTML format (3)" to_port="file"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
</process>
  </operator>
</process>

While this works really great, it does have one snag. If you want to display multiple stockcharts, you’d have to figure out a way to update the location of the chart relative to where your blog post is.

This is important if you show more than one post per page. When I generated two posts with two different stockcharts, the HTML made them relative to the top of the page. In other words, the one chart was on top of the other one. I’m sure this is an easy fix but something I’m not going to bother with for this tutorial!


  1. Nope, got it to work in Markdown with the help of https://twitter.com/lllIIlIlIl



Blogging tutorials RapidMiner


Previous post
Today's Chart: MCD The chart of $MCD is generated using the Plot.ly JS library and RapidMiner Studio. This is a daily chart from 2015-10-18 to 2016-04-18. For further
Next post
Today's Chart: FBIO The chart of $FBIO is generated using the Plot.ly JS library and RapidMiner Studio. This is a daily chart from 2015-10-18 to 2016-04-18. For further