Rusty Finger Blog

I blog because I can’t remember stuff. = P

Hackathon-dot-Taipei

Has been exactly a year since I last joined a community event. I was honned on building the next Facebook!

… not really, but something interesting for sure. = )

Data Warehouse Terminologies

Our company has a pretty crappy reporting service. It was trying to denormalize data from the main database and storing them into a MongoDB. The catch is, data from the original database was still kept relatively independent tables and relationship maintained between them. The result are very slow queries and unreliable data transport, we are not using our tools correctly.

So the hunt for a better reporting service was on! And some really good articles came up, with some very important terminology.

Database as a Fortress

http://dan.chak.org/enterprise-rails/chapter-4-as-a-fortress/

So this article came up. It was the perfect! Filled with plenty of terminology and very informative. Some of the stuff came up were:

OLTP (Online Transaction Processing) - Highly normalized data, optimised for store, update and retrieve data. Almost all websites are OLTP.

OLAP (Online Analytical Processing) - Geared toward getting meaningful business knowledge out of the data sets. Ad-Hoc rules such as quarterly, weekly informations are usually stored for quick access.

The difference between OLTP and OLAP makes sharing of database a bad idea. Hence, MySQL slave is also out of the question.

Data Warehouses and MultiDimensional Data Analysis

https://www.youtube.com/watch?v=mURhRy9plrY

This talk was given on RailsConf 2015, it goes deeper into the topic and provider further insights.

Dimensional Modeling - Treat different criterias in a query as different dimensions (ie. time, location, product), the measurement you are looking for is consider as facts. Facts then become the intersections of different dimensions.

IMG_20141221_173200

This is such a powerful concept! Facts then become this flexible concept, which can be cached and create on the fly!

The schema for this kind of database are usually call Star or Snowflake. Center being the facts you are asking the OLAP system, and branches being the dimensions.

Within some dimensions, there might be one or several hierarchies. For example, time dimension might have a default hierarchy of Day, Month, Quarter, Year. Each level of the dimension would be related and/or depending on it’s predecessor. With that in mind, we might want to build another hierarchy within the time dimension. It might be a weekly report and the dimension become Day, Week, Year.

This hierarchy is then stored as grouping or classes as an extension of the dimension table. Your Star schema is growing to become Snowflake schema!

The talk went on and suggested some tools for implementing this idea.

  • OLAP Technologies - Pentaho, Mondrian, Mondrian-olap(JRuby)

    They focus on efficient OLAP queries. Also provides easy store calculation formulas, which can provide calculation result from the columns and be easily query-able. Great for building Ad Hoc queries.

  • ETL Process - ETL(Ruby), Kiba(Ruby)

    Single threaded ETL(Extraction-Transform-Load). Have easily configurable DSL. However, you should consider a multi-threaded ETL if you have to parse high volume of data in a short period of time.

  • Analytical Relational Database - MonetDB, Vertica, InfoBright, Redshift

    SQL-like syntax, optimised for analytical queries. This is achieved by using Columnar Storage, virtually it is the same as using tables. But physically data are organised in columns. (The opposite is Row-based Storage, which DB such as MySql and PostgreSQL uses)

HOW TO BUILD A DATA-WAREHOUSE IN 4 WEEKS

http://victorsavkin.com/post/9209987806/how-to-build-a-data-warehouse-in-4-weeks-part-1

This seems to be a very detailed step by step walk through on building a data warehouse, I have yet to take a closer look.

There IS ALSO THE DW BIBLE

http://www.amazon.com/gp/product/1118530802

Yes, one of those days… I’ll get around to it.

It is Warehouse Time!

Some of our colleagues have set out on the quest to build a new and better analytical system. This post is acting as a cheatsheet for me to quickly recap on the important terminologies on such a system. Hopefully you find it useful too.

Cheers

Some Thoughts After Few Months of Freelancing

IMG_20141221_173200

So, I finally got in to freelancing and roll with it for a few months, the result?

There’s Plenty of Jobs, But Not Enough Dough

And I learned plenty the hard way.

Read that EFF-ing Project Documentation

I am optimistic by nature, it gives me confidence, and … blinds my judgement. Hence I end up rebuilding things many times over. The second project was a redesign of their original WP catalog site, it was pretty run down even for Wordpress standards. So I was getting lazy and wanted to cut corners by implement yet another Wordpress, and PROFIT!

No,

it turned out the function they want was a pain to build in Wordpress. Zero PHP experience and lack of prior knowledge with the framework didn’t help here. It was a month into the project for me to toss out everything and rebuild the project in Rails. If I had studied the customer criterias, I could have moved on to the next project already.

Leave no imagination space

For the first project (which I regretted ever took parts in… took up three months of my solid work hours, with ZERO returns) I was handed a zip with bunch of assets, and told “get it to run in rails”.

It turned out the htmls and assets was not meant for implementation in Rails. I was stupid to think it was simple and easy to implement.

They left big chunks of JS actions and CSS adjustment for me to do. And I was nowhere near the agreed schedule.

It looked mighty bad on my part, and rightfully so. It was not a project for me from the very beginning.

Not everybody wants the “Best Practice”

Rails enforced a series of best practices for building the web. RestFUL, Assets Precompiling, CSRF Tokens… just to name a few. It is great to quickly build reliable and secure web sites.

But… Not everybody wants it…

Yes, they just want it fast, and undercuts you. so yeah….

Work Fast

Speed is of the Essence

In the world of the freelancing, speed is everything. You don’t deliver, you don’t get paid.

Capisci?

Say No

Time is your biggest and only assets. - Somebody

So learn to say no will save you plenty of life and your sanity. and while you are on “saying no training wheels”, you are going to feel like an idiot plenty.

During the process of freelancing, you really learn to value your time and energy. Thinking too much about the money, you will get burn outs very quickly. You eventually learn the price tag for your line of skill. It is helpful for future employment and negotiation for cases.

Conclusion

If you want(need) something, Get rid of it. - Glenn Beck

Came across this quote and hold it as guidelines when I need to make a decision. When you think you need something that badly, you really need to take a step back and think about what you are willing to trade for your want/need. Anywho, until next time~ = )

ps. LOL It is Nov.4th now. took me 6 months to finish this post… need to get my acts together.

Quick Notes on Linking Woocommerce With Shop.com

Disclaimer I have 0 experience with php, so yeah…

So I’ve been busy with mutiple cases since the start of 2015, and just want to leave a quick note on what I’m currently working on.

Enter Shop.com

They are just another direct sale company, whose philosophy I completely disagree with.

Option Simple-And-Easy: Store Exporter Delux

They create costum order fields, and export! That simple!! They even got an url for cron jobs to call!!!! Wow

Cost: USD $20

Wordpress and function.php

Being a wordpress newb and all, I wasted good amount of time trying to write code in respective locations. eg, header.php for getting RID, display RID in form, etc.

But it was all so much simpler when you follow the WooCommerce conventions, and do some RTFM. *Sad Face*

So in each WP theme you installed, contains a function.php, where all the WP/WC hooks and filters live. Hook performs functions, filter alters display. WC Hook/Filter Reference

Great!! One file to RULE THEM ALL

So this is what I finally come up with.

Get RID:

function set_rid() {
  // echo $_GET["RID"];
  setcookie ("RID", $_GET["RID"]);
  // echo $_COOKIE["RID"];
}
add_action('init', 'set_rid');

Add RID Field on Checkout:


add_action( 'woocommerce_after_order_notes', 'my_custom_checkout_field' );

function my_custom_checkout_field( $checkout ) {
  if( isset($_COOKIE["RID"]) ) {
    $RID = $_COOKIE['RID'];
    woocommerce_form_field( 'RID', array(
      'type'          => 'text',
      'class'         => array('my-field-class form-row-wide'),
      'label'         => __('美安Shop.com 編號'),
      'default' => $RID,
    ), $checkout->get_value( 'RID' ));
  }
}

Updates the order meta with RID


add_action( 'woocommerce_checkout_update_order_meta',
  'rid_checkout_field_update_order_meta' );

function rid_checkout_field_update_order_meta( $order_id ) {
  if ( ! empty( $_POST['RID'] ) ) {
    update_post_meta( $order_id, 'RID', sanitize_text_field( $_POST['RID'] ) );
  }
}

That is it!! = )

Now just use Store Exporter to fish out the data, and you are set!!

Option Hardwork: Do-It-My-Self

F*** This…

too damn annoying constructing sql string for getting key-value pair.

I’ve been bigging through Wordpress and Woocommerce for a bit this afternoon, this are what I gathered.

  • Both of this systems stores most informations in key-value pair
  • Items are found in _posts and _postmeta
  • Orders are found in _woocommerce_order_items and _woocommerce_order_itemmata
  • wc_add_order_item_meta()adds attributes to order_itemmata
  • now I just need to construct the sql to fetch all the stuff I need.

HanGee Hackathon Dec 2014

IMG_20141221_173200

I finally got the chance to attend my first hackathon last weekend(21st, 22nd)! Two days, free food, pizza party, and intense geeky coding all through the night. Couldn’t get much more interesting than this!

I saw the even the day before, with wife and kids out of town, it couldn’t have been on a better timing. I was exposed to NodeJS and ReactJS during the first day by smlsunxie and coodoo. The topic was on NodeJS Testing and Beginning with ReactJS respectively. Coodoo’s speak and codes was instrumental for our project. We did an project which I had always wanted to do, location-based-reddit-like-service.

So, I’ll try to summarized what I come to understand about ReactJS, an awesome JS framework.

Let’s start somewhere simple.

Backend

The backend is rather straight forward, install strongloop, and

slc loopback

will generate backend scaffolding.

slc loopback:model

for triggering the interactive model creation.

Finally, you do

slc run.

to get your backend server running.

Strongloop/loopback is a basic Node API server, following the REST conventions. However, unlike Rails, which has 7 routes (8 to be precise), Strongloop creates 11 routes for you.

During the hacking process, initially, we wanted to place our distance calculations inside backend. However, after much thought, I was convinced otherwise and did not touch the backend code besides the system generated models.

The reason?

If I understand correctly, MVW Frameworks and Client-Side WebApp encourages developer to place much of the calculations on the client side. In our case, it felt right! After coming to this conclusion, our development also make so much more sense. No more, “pass x to server, pass back y, then calculate x - y to display z”, it was all in one simple and easy to understand module. Life’s Good. *Thumbs Up*

Frontend

This is where ReactJS comes in, he also brought along he’s less popular brother, Flux.

Above image sums up everything React and Flux is doing, keep it in mind and you are good. Here’s how it kept us sane.

React organizes views into .jsx files, which interestingly collects both html-like dsl and javascripts of the same behavior together. The user then triggers actions which are captured and passed along to the action creator, which then communicates with the backend and/or sends the action object to dispatcher. Dispatcher is responsible for controlling the flow, and will wait for the previous process to finish before allowing the next one. Store is where the business logic are placed, and it will act on the data passed along from the dispatcher, then emit a request to update the virtual DOM of the React view.

phew, that’s a mouthful.

But once you get the basic concept, development was a breeze. Everything work as they should, and you can bring teammates up to speed pretty rapidly. One thing could have done better, instead of distance calculation in view, we should have placed it in store.

IMG_20141221_173804

It was a really awesome event! Really good to know people of the similar interest. One thing that surprised me, was the Rails background I had was pretty helpful, much of the concept can be easily translated.

Anyway, Happy Christmas Everybody!! = )

IMG_20141221_194013

Finally Got My Desk Working Properly

I have been reading about working standing up since two years ago, finally stumble upon an affortable solution. This is what my desk looks like now.

IMG_20141012_154356

Here is the original post.

So for any Taiwan folks reading this, here’s a list of things I bought:

Ikea item list

  1. LACK 邊桌 $245 901.616.73
  2. EKBY JÄRPEN 層板 $395 301.814.95
  3. EKBY VALTER 支撐架 $50 X 2 401.674.89
  4. Some bare screws

All of the items add up to about $740NTD. I got 2 tables for my dual screens, and bigger work space is really wonderful. I wished I had taken more pictures during the assembly, but it was pretty straightfoward.

After working without a chair for almost a month, my butt feels much tighter, and my legs have become stronger. However, I do fatigue much faster during my kung-fu class, it would probably improved after I am more used to the new workspace.