My Chinese Big Brother


One of my Asian friends recently posted a link to a terrifying use of Machine Learning. This is what I call the “dark side” of this field, the use of machine learning by a government to make you behave a certain way.

1984’s Big Brother

China is building its own version of 1984’s Big Brother, a massive scoring system that’s probably a large scale classification algorithm most likely sitting on top of a Big Data structure like Hadoop. I call this an utter and complete abuse of machine learning.

The government hasn’t announced exactly how the plan will work — for example, how scores will be compiled and different qualities weighted against one another. But the idea is that good behavior will be rewarded and bad behavior punished, with the Communist Party acting as the ultimate judge.

Yes, we use classification and scoring system in everyday commerce. Banks use it to grade your credit card worthiness. Businesses use it to determine your propensity to buy their product. This project goes beyond all that.

Social Trust

China wants to learn if you say and do anything to break “social trust.”

They will scrap online sources available and assign you a score. This score will determine your trustworthiness. How they define trustworthiness remains the question.

This can manipulated and gamed. Questionable data quality can lead to misclassification and mistakes and Chinese people will suffer those consequences.

IMHO China’s drive for a harmonious society with a “Asian Big Brother” will only hasten it’s demise.

Adding EURUSD Historical Volatility Predictions

I recently connected my historical volatility prediction processes to the FRED API and pulled in closing data for the Euro. I want to autogenerate currency volatility predictions just like what I do for the stock indices.

The only problem now is that the training and optimization part takes to long so I have to start breaking up the training/optimization and the auto post autogreneration processes. Plus I’m starting to hit the limit of my laptop and will likely move my training/optimization to a RapidMiner Server on AWS.

On top of all this, I should build a process that will tally how correct the predictions are and then autopost that!

Lots of work to do but it’s damn cool!


I just posted how I did it over at the RapidMiner blog. It’s quite easy and fast.

Rebuilding a Blog – Part 4

This post on rebuilding a blog is continuation of my previous post.

In this post I wanted to review some goals I created in Google Analytics. These goals were created to see how people are interacting with my site.


I created a total of 7 goals:

The chart shows some interesting results. I have any amazing amount of sticky time and people read more than 1 post. This makes me think I should abandon Google Ads and sell some space on my own.

On top of this, there’s some activity in the use of the search box and people do click on the Tutorial tags. No one cares about the SEO, and RSS.


Although this will be deprecated soon my Google, I do like to look at this. It’s nice to see that all things are green and there is some activity in new users and page sessions.

Overall, things are improving here but the lack of tag click – except the Tutorials tag – leads me to guess that:

  1. People don’t care about those tags, or
  2. People don’t know enough about navigating to those tags.

Next Steps

I think I will make some small navigation changes to allow users to search via tags OR via the search box. Next post I’ll examine my top read content and determine which articles need to be refreshed.”


Rewiring Education

Last night the kids and I were across the street at another family’s house for dinner. Their kids are the same age as my daughter and son and often stay for dinner at our house. Last night after the dinner we got to talking about the current state of education and how we can better educate our kids. Both the husband and wife have careers in the creative fields and are big supporters of STEM.

The conversation got me thinking about the role of parents in guiding their children in education but also how technology changes the game.

My wife and I religiously go to Back to School Night” and attend every teacher conference. We are in routine contact with the teachers through email and we give our kids plenty of secondary education as well (Chinese and German language classes).

My daughter just started 6th grade and she was excited and nervous to finally go to a big” school. We visited her school and met all her teachers during the Back to School Night a few weeks ago. Now she has to navigate the hallways to get to her classes and remember her locker combination. She has all the normal classes like French, Math, Science, Social Studies and PE, but there subtle and interesting differences than when I was in school.

Of course I didn’t expect things to stay the same from when I was in 6th grade – I expected some technological enhancements – but what I didn’t expect was just how intertwined things got.

Her French teacher emails her twice a week with a link to online lesson plans. There she can access French-English word flash cards, listen to an mp3 sound out the French word, and take quizzes” to guage her progress. She has the standard math textbook but it stays home and is used for reference. Her math homework is done online and submitted to her teacher. She has three tries to get the answer right and then it records it.

She even has a Research class that meets once a week where the students learn how to use the Internet to research, evaluate, and properly reference material.

The funny thing about this? She doesn’t complain about doing her homework. She accesses her work online through her iPhone and does her homework. She takes to education enabled by technology like a fish to water.


What my daughter doesn’t realize is that her education is slowly becoming customized. While the average American complains about Common Core and how it doesn’t make sense to them, there are entrepreneurs figuring out ways to customize learning to each child.

Children and adults all learn at difference paces and in different ways. Some people and children are visual learners, others auditory, and some learning by doing (tactile learning). Why, then, should the one sized fits all” method of education be applied all children?

While I’m not an expert in education, I do believe that everyone can learn and have a rich education. The hard part is figuring out how each child learns and I believe the computer is the way forward.

Auditory learners can easily access podcasts or mp3′s to listen to. Computers are perfect ways for visual learners to access material, and tactile learners can just start writing code or building projects with a Raspberry Pi or Ardunio.

Now, more than ever, online courses are available for free. You can access materials on Python Programming, Anthropology, Art, Humanities, Data Science, anything you can dream of. High School students can access these college course and get a headstart on higher learning.” Our High School offers online courses to its students that want to learn a particular course BUT wouldn’t be available because they couldn’t fill an entire classroom with students.

This is the key.

Education enabled by technology frees you from the constraints of filling up a classroom with students just to justify the courses existence! Now you can have a class filled with students, each one of them learning something different specifically geared for them.

We truly live in amazing times.

Source: Blot NMT Feed

Latest Writings Elsewhere for September 2016

Just a quick list of the content I’ve created some place other than this blog. This current list is 100% RapidMiner related but I’d like to branch out into guest posting. If any of my readers would like a guest post on their blog or site from me, then contact me (tom %at% neuralmarkettrends %dot% com).

Source: Blot NMT Feed

A Twitter Bot in Groovy Script – Part 2

Just a quick update

I’m trying to integrate my R2D2 Twitter Bot using Groovy Script into RapidMiner but I’ve run into problems. Originally I tried to copy and paste the groovy script from my last post directly into the Execute Script operator in RapidMiner (v7.2) but when I executed it, it failed. It gave me Classpath errors and problems calling the TwitterFactory Class.

I had installed the Twitter4j JAR file into my /lib directory and checked all my environmental path configurations. I was going nuts until I asked Helge from our Support group.

It turns out that many Java related security enhancements were made in RapidMiner v7.2. Most notably was the disabling of reflection” which is used by the TwitterFactory class in Twitter4j. This is what cause it blow up when I executed it.

Helge suggested to try it with version 7.1, and I did. It worked perfectly.

So what to do now? Either I build the entire Twitter Bot by hand in Groovy or I just go back to my Python one.


Software Advice Interview

Just a short note, but Tom Wentworth and I were interviewed by Software Advice. They were evaluating easy to use but powerful Text Analytics tools for small to medium sized business.

RapidMiner is the easiest to use and most fully featured text mining tool of the platforms I demoed. With the AYLIEN extension, you’ll be able to perform basic sentiment analysis within minutes of downloading and installing. Link

Source: Blot NMT Feed