Step-by-step

1. Download and install the HDP Sandbox

Download the latest (2.3 as of this writing) HDP Sandbox here. Import it into VMware or VirtualBox, start the instance, and update the DNS entry on your host machine to point to the new instance’s IP.

On Mac, edit /etc/hosts, on Windows, edit %systemroot%\system32\drivers\etc\ as administrator and add a line similar to the below:

  1. 192.168.56.102 sandbox sandbox.hortonworks.com

2. Download and install the latest NiFi release

Follow the directions here. These were the steps that I executed for 0.4.1

  1. cd /tmp
  2. wget http://apache.cs.utah.edu/nifi/0.4.1/nifi-0.4.1-bin.zip
  3. cd /opt/
  4. unzip /tmp/nifi-0.4.1-bin.zip
  5. useradd nifi
  6. chown -R nifi:nifi /opt/nifi-0.4.1/
  7. perl -pe 's/run.as=.*/run.as=nifi/' -i /opt/nifi-0.4.1/conf/bootstrap.conf
  8. perl -pe 's/nifi.web.http.port=8080/nifi.web.http.port=9090/' -i /opt/nifi-0.4.1/conf/nifi.properties
  9. /opt/nifi-0.4.1/bin/nifi.sh start

3. Create a Solr dashboard to visualize the results

Download a new Solr dashboard, start the service, and create a new collection to store stock price changes:

  1. export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
  2. wget https://raw.githubusercontent.com/vzlatkin/Stocks2HBaseAndSolr/master/Solr%20Dashboard.json -O /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/default.json
  3. /opt/lucidworks-hdpsearch/solr/bin/solr start -c -z localhost:2181
  4. /opt/lucidworks-hdpsearch/solr/bin/solr create -c stocks -d data_driven_schema_configs -s 1 -rf 1

4. Create a new NiFi flow to pull from Google Finance API, transform, and store in HBase and Solr

Solr is used for indexing the data, Banana UI is used for visualization, and HBase is used for future-proofing. HBase can be used to further analyze the data from Storm/Spark or to create a custom UI. The get the data into these tools, follow the steps below:

  • Start HBase via Ambari
  • Create a new table:
    1. hbase shell
    2. hbase(main):001:0> create 'stocks', 'cf'
  • Then download this NiFi template to your host machine.
  • To import the template, open the NiFi UI
  • Open Templates manager:

https://community.hortonworks.com/storage/attachments/1144-import-into-nifi-step-1.png

  • Find the template on your local machine and import it:

https://community.hortonworks.com/storage/attachments/1128-import-into-nifi-step-2.png

  • Drag and drop to instantiate a new template:

https://community.hortonworks.com/storage/attachments/1130-import-into-nifi-step-3.png

  • Double click the new process group:

https://community.hortonworks.com/storage/attachments/1141-import-into-nifi-step-4.png

  • You'll need to enable the HBase shared controller. To do so, click the right mouse button over the "Send to HBase" process, then click "Configure", then "Properties" and the "Go to" arrow to access the controller. Finally, click the "Enable" button.

https://community.hortonworks.com/storage/attachments/1142-import-into-nifi-step-5.png

  • Now start all of the processes. Hold down the Shift-key, and select all of the processes on the screen. Then click the start button:

https://community.hortonworks.com/storage/attachments/1143-import-into-nifi-step-6.png

You should see a flow that looks like the below screenshot

https://community.hortonworks.com/storage/attachments/1145-nifi-screenshot.png

The reason for so many processes is that the response from Google Finance API needs to be transformed. First, we remove the comment characters '//' from the response. Second, we split the array into individual JSON objects. Third, we extract the relevant attributes. Fourth, the timestamp has the format of UTC, but it is actually in EST timezone, therefore, we fix that. Finally, we send the information to HBase, Solr, and the NiFi bulletin board for logging.

results matching ""

    No results matching ""