Friday, December 20, 2013

RESTful Java Client for Apache HttpClient

Apache HttpClient is a robust and complete solution Java library to perform HTTP operations, including RESTful service. This blog post shows how to create a RESTful Java client with Apache HttpClient, to perform a “GET” and “POST” request.

Apache HttpClient is available in Maven central repository, just declares it in your Maven pom.xml file.

1
2
3
4
5
<dependency>
 <groupId>org.apache.httpcomponents</groupId>
 <artifactId>httpclient</artifactId>
 <version>4.1.1</version>
</dependency>

Following is the way to implement GET request.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
public HttpResponse doGet(DefaultHttpClient httpClient, String resourcePath, String userName, String passWord) {
 try {
     HttpGet getRequest = new HttpGet(resourcePath);
     getRequest.addHeader("Content-Type", "application/json");

     String userPass = userName + ":" + passWord;
     String basicAuth = "Basic " + javax.xml.bind.DatatypeConverter.printBase64Binary(userPass.getBytes("UTF-8"));
     getRequest.addHeader("Authorization", basicAuth);

     httpClient = (DefaultHttpClient) WebClientWrapper.wrapClient(httpClient);

     HttpParams params = httpClient.getParams();
     HttpConnectionParams.setConnectionTimeout(params, 300000);
     HttpConnectionParams.setSoTimeout(params, 300000);

     HttpResponse response = httpClient.execute(getRequest);

     return response;
 } catch (ClientProtocolException e) {
     e.printStackTrace();
     return null;
 } catch (IOException e) {
     e.printStackTrace();
     return null;
 }
}

Following is the way to implement POST request.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
public HttpResponse doPost(DefaultHttpClient httpClient, String resourcePath, String jsonParamString, String userName,
                       String passWord) throws Exception{
 try {
     HttpPost postRequest = new HttpPost(resourcePath);

     StringEntity input = new StringEntity(jsonParamString);
     input.setContentType("application/json");
     postRequest.setEntity(input);

     String userPass = userName + ":" + passWord;
     String basicAuth = "Basic " + javax.xml.bind.DatatypeConverter.printBase64Binary(userPass.getBytes("UTF-8"));
     postRequest.addHeader("Authorization", basicAuth);

     httpClient = (DefaultHttpClient) WebClientWrapper.wrapClient(httpClient);

     HttpParams params = httpClient.getParams();
     HttpConnectionParams.setConnectionTimeout(params, 300000);
     HttpConnectionParams.setSoTimeout(params, 300000);

     HttpResponse response = httpClient.execute(postRequest);

     return response;
 } catch (ClientProtocolException e) {
     throw new ClientProtocolException();
 } catch (ConnectException e) {
     throw new ConnectException();
 }
 catch (IOException e) {
     e.printStackTrace();
     return null;
 }
}

You can used WebClientWrapper class to ignore the certificate. You can find it from following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

import org.apache.http.client.HttpClient;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.impl.client.DefaultHttpClient;

public class WebClientWrapper {

    public static HttpClient wrapClient(HttpClient base) {
        try {
            SSLContext ctx = SSLContext.getInstance("TLS");
            X509TrustManager tm = new X509TrustManager() {
                public void checkClientTrusted(X509Certificate[] xcs,
                                               String string) throws CertificateException {
                }

                public void checkServerTrusted(X509Certificate[] xcs,
                                               String string) throws CertificateException {
                }

                public X509Certificate[] getAcceptedIssuers() {
                    return null;
                }
            };
            ctx.init(null, new TrustManager[] { tm }, null);
            SSLSocketFactory ssf = new SSLSocketFactory(ctx);
            ssf.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
            ClientConnectionManager ccm = base.getConnectionManager();
            SchemeRegistry sr = ccm.getSchemeRegistry();
            sr.register(new Scheme("https", ssf, 443));
            return new DefaultHttpClient(ccm, base.getParams());
        } catch (Exception ex) {
            return null;
        }
    }
}


Java Code Geeks

Sunday, December 1, 2013

WSO2 Complex Event Processor Integration for Apache Stratos 4.0.0

In this hangout I'm demonstrating Complex Event Processor (CEP) integration for Apache Stratos 4.0.0 architecture. In this hangout I'm talking about,
  • Introduction to CEP
  • Why CEP ?
  • CEP Integration
    • Integration Architecture
    • Event Flow
  • How to write CEP configs
    • Stream definition
    • Input event adapter
    • Event builder
    • Execution plan
    • Event formatter
    • Output event adapter
  • Demonstrations 


Monday, October 14, 2013

Apache Stratos Hangout VII - Testing Apache Stratos with OpenStack

At the moment, we are in the release of Apache Stratos 3.0.0 Incubation RC4. In this hangout I'm going to demonstrate how to test RC4 packs with OpenStack.
  • How to configure stratos setup
  • Run the stratos setup and install apache stratos packs
  • Start the packs
  • Create a tenant
  • Subscribe to mysql cartridge 
  • Subscribe to php cartridge
  • Demonstrate wordpress sample application

Tuesday, October 1, 2013

Configure OpenSSO for WSO2 Business Activity Monitor

In this blog post you can find how to configure OpenSSO for WSO2 Business Activity Monitor.

First of all you have to download WSO2 Business Activity Monitor and OpenAM. You can download WSO2 Business Activity Monitor from here and OpenAM from here. Make sure to download the war file for OpenAM. Also make sure to get standalone tomcat rather using tomcat by sudo apt-get install tomcat7.

As the first step you have to deploy the OpenAM webapp in the tomcat. It's easier to rename OpenAM war file as openam.war and deploy in tomcat.

After that you have to restart the tomcat. Then you can configure openam by accessing http://localhost:8080/openam

I have tried with default configurations.

After configure OpenAM, you have to create the Hosted Identity Provider by clicking on Hosted Identity provider button in Common Task Tab.


You can provide what ever name you wanted. Here I put as openamIDP and Singing key as test and circle of trust as sample.


After that you have to register Remote Service Provider. You can do it by uploading SP.xml file. Following you can find sample SP.xml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<EntityDescriptor entityID="carbonServer" xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
    <SPSSODescriptor AuthnRequestsSigned="false" WantAssertionsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
        <SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://localhost:9443/acs/fedletSloRedirect" ResponseLocation="https://localhost:9443/acs/fedletSloRedirect"/>
        <SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://localhost:9443/acs/fedletSloPOST" ResponseLocation="https://localhost:9443/acs/fedletSloPOST"/>
        <SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:SOAP" Location="https://localhost:9443/acs/fedletSloSoap"/>
        <NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified</NameIDFormat>
        <AssertionConsumerService index="0" isDefault="true" Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://localhost:9443/acs"/>
        <AssertionConsumerService index="1" Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Artifact" Location="https://localhost:9443/acs"/>
 </SPSSODescriptor>
 <RoleDescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:query="urn:oasis:names:tc:SAML:metadata:ext:query" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="query:AttributeQueryDescriptorType">
 </RoleDescriptor>
 <XACMLAuthzDecisionQueryDescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol" WantAssertionsSigned="false" />
</EntityDescriptor>

You have to provide same circle of trust for Remote Service Provider as well.

After that you have to do some more configurations for Hosted IDP and the Remote Service Provider. To do that you have to click on Federation tab and select the created IDP. Remove all the NameID value Map items under the created IDP and insert the following to the created IDP's NameID value Map and save it.

"urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified=cn"


After that you have to configure Remote Service Provider. Click on the remote service provider created by you and tick on Authentication Requests Signed, Assertions Signed and Post Response Signed and save it.


Now you have finish the configurations of OpenAM. Now you have to configure the WSO2 Business Activity Monitor. To do that open the authenticators.xml in <BAM_HOME>/repository/conf/security and do the following changes to SAML2SSOAuthenticator section.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<!-- Authenticator Configurations for SAML2SSOAuthenticator -->
<Authenticator name="SAML2SSOAuthenticator" disabled="false">
  <Priority>10</Priority>
  <Config>
    <Parameter name="LoginPage">/carbon/admin/login.jsp</Parameter>
    <Parameter name="ServiceProviderID">carbonServer</Parameter>
    <Parameter name="IdentityProviderSSOServiceURL">http://localhost:8080/openam/SSOPOST/metaAlias/idp</Parameter>
    <Parameter name="IdPCertAlias">opensso</Parameter>
  </Config>
</Authenticator>

Now you have successfully configured OpenAM and WSO2 BAM. As the final step you have to export the OpenAM certificate to WSO2 BAM. You can do it by using Java keytool.
  • You can find OpenAM public key in /home/openam/openam/keystore.jks
  • Here we will be using the default shipped openSSO keystore certificate. It has the alias name of ‘test’. The default password is ‘changeit’. To export the public key of ‘test’ you can use "keytool -export -keystore keystore.jks -alias test -file test.cer"
  • The public key will get stored in ‘test.cer’ file. you can view the certificate content with the command, "keytool -printcert -file test.cer"
  • Now import the ‘test.cer’ into Carbon key stores found under <BAM_HOME>/repository/resources/security/wso2carbon.jks by using  "keytool -import -alias opensso -file test.cer -keystore wso2carbon.jks". Password is wso2carbon.
  • You can view the imported certificate using the command  "keytool -list -alias opensso -keystore wso2carbon.jks -storepass wso2carbon"
Now you are ready to test. Try accessing the carbon management console. (https://localhost:9443) The call will redirect you to IDP (openSSO login page). Enter username and the password in the openSSO login page. Once you properly authenticated you will redirect back you to the WSO2 Carbon product login page as a logged in user. Please note: The authenticated user has to be in the Carbon servers’ user-store for authorization (permission) purposes. Since the above described test environment does not share the same user store between IDP (openSSO server) and SP (Carbon server) i created a user called ‘amAdmin’ in Carbon server user store. Otherwise there will be a authorization failure during the server login.

Monday, September 23, 2013

Configure WSO2 Business Process Server (BPS) with JBOSSMQ

The following instructions describe how to set up the JMS transport with JBossMQ, the default JMS provider in JBoss Application Server 4.2

To configure the JMS transport with JBossMQ:

You can download JBossMQ from here.

Copy the following client libraries to the <BPS_HOME>/repository/components/lib directory.
  • <JBOSS_HOME>/lib/jboss­system.jar
  • <JBOSS_HOME>/client/jbossall­client.jar
Enable the JMS transport listener by adding the following listener configuration to the <BPS_HOME>/repository/conf/axis2/axis2.xml file:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
<!--­­ Configuration for JBoss 4.2.2 GA MQ ­­-->
<transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener">
  <parameter name="MyQueueConnectionFactory" locked="false">
    <parameter name="java.naming.factory.initial" locked="false">org.jnp.interfaces.NamingContextFactory</parameter>
    <parameter name="java.naming.factory.url.pkgs" locked="false">org.jnp.interfaces:org.jboss.naming</parameter>
    <parameter name="java.naming.provider.url" locked="false">jnp://localhost:1099</parameter>
    <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">/ConnectionFactory</parameter>
    <parameter name="transport.jms.Destination" locked="true">queue/susaQueue</parameter>
  </parameter>
  <parameter name="default" locked="false">
    <parameter name="java.naming.factory.initial" locked="false">org.jnp.interfaces.NamingContextFactory</parameter>
    <parameter name="java.naming.factory.url.pkgs" locked="false">org.jnp.interfaces:org.jboss.naming</parameter>
    <parameter name="java.naming.provider.url" locked="false">jnp://localhost:1099</parameter>
    <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false">/ConnectionFactory</parameter>
    <parameter name="transport.jms.Destination" locked="true">queue/susaQueue</parameter>
  </parameter>
</transportReceiver>

Enable the JMS transport sender by uncommenting the following line in the <BPS_HOME>/repository/conf/axis2/axis2.xml file:


1
<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>

After that you have to create the queue in JBOSS. To do that create a file called myQueue-service.xml in <JBOSS_HOME>/server/default/deploy and insert following lines to it.


1
2
3
4
5
6
<server>
 <mbean code="org.jboss.mq.server.jmx.Queue"
    name="jboss.mq.destination:service=Queue,name=susaQueue">
   <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends>
</mbean>
</server>

Now you can start the JBOSS server. You can ensure that the queue which you are created inside JBOSS by reading the log file.

Start the BPS and ensure that the logs prints messages indicating that the JMS listener and sender are started and that the JMS transport is initialized.

Thursday, August 29, 2013

Create OpenStack Tomcat Cartridge in Apache Stratos

Cartridges

A Cartridge is a package of code or configuration that plugs into Stratos to offer a new PaaS Service. A Cartridge is also a Virtual Machine (VM) image plus configuration and it can operate in two modes that are namely single tenant and multi-tenant.

For more information see Cartridges.

Before create the cartridge you have to make the OpenStack image.

Create OpenStack image to support Tomcat
  • First you have to download ubuntu cloud image and upload it to OpenStack. You can download ubuntu cloud images from here.
  • After that you have to upload that image to OpenStack. Please refer creating OpenStack images for more details.
  • After successfully upload the cloud image to OpenStack you can launch instance from that image.
  • Now you need to setup the instance for Tomcat. To do that you have to install following software to that instance. You can do it by ssh to that instance.
    • java 1.6.x - refer here for more details
    • tomcat7
    • git
    • ruby
    • thrift - refer here for more details
    • xml_grep - use sudo apt-get install xml-twig-tools
  • Installing tomcat7 will install several java versions. But to this installation you need java 1.6.x. To set this as default you can use following commands.
    • sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.6.x/jre/bin/java" 1
    • sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.6.x/bin/javac" 1
    • sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.6.x/jre/bin/javaws" 1
  • After installing above prerequisites, you have to update /opt directory with following things.
    • get-launch-params.rb
    • healthcheck.sh
    • stratos-init.sh
    • cartridge_data_publisher_1.0.2
  • You can get above from apache stratos git repo.
  • After completing this your instance is ready. Now you can create the snapshot from that instance and make the image from that snapshot. Before creating the image you have to update /etc/rc.local file to trigger stratos-init.sh at the boot up time. For more details please refer here to get an idea about how to create OpenStack images.
Create Tomcat Cartridge
Now you are in the final stage of the Tomcat cartridge creation in apache stratos. Simply you can refer here to install apache stratos. After successful installation you can edit the tomcat.xml according to your IaaS image properties. You can download sample tomcat.xml file from here.

After doing that you can start Apache Stratos Controller and subscribe to the created Tomcat cartridge.

Wednesday, August 21, 2013

Install Apache Stratos

Apache Stratos could be installed on a single node or on multiple nodes. If you are installing on multiple nodes copy stratos_installer to each node and update configuration parameters in conf/setup.conf accordingly.

Installation Guide
1. Install following prerequisites. Use sudo apt-get install for this.
  • java    -jdk1.6.x
  • git
  • facter
  • zip
  • mysql-server
  • Gitblits 

2. Build Apache Stratos from source:
  • git clone https://git-wip-us.apache.org/repos/asf/incubator-stratos.git
  • cd incubator-stratos
  • mvn clean install

3. Copy cloud controller, stratos controller, elb, agent and cli packages to a folder inside stratos_installer. May be this could be called "stratos_installer/packages":
  • apache-stratos-cc-<version>.zip 
  • apache-stratos-sc-<version>.zip
  • apache-stratos-elb-<version>.zip   
  • apache-stratos-agent-<version>.zip
  • apache-stratos-cli-<version>.zip 
4. Download WSO2 Message Broker binary distribution from http://wso2.com and copy it to stratos-pack-path. Here you could use any preferred message broker product which supports AMPQ.

5. Extract WSO2 Message Broker distribution to desired path (this will be identified as stratos-path) and set it's port offset in repository/conf/carbon.xml to 5. This will set the actual port to 5677.

6. Download MySql Java connector from http://dev.mysql.com/downloads and copy the jar file to the above packages folder.

7. Create and download the keys from IaaSs and store them on a secure location.

8. If Apache Stratos being setup in multiple nodes open up the security rules in IaaSs for the following ports (defined in ./conf/setup.conf) 22, 443, 8280, 4103, 4100, agent_https_port, agent_http_port, elb_port, agent_clustering_port, sc_cluster_port, elb_cluster_port, cassandra_port, stratos_db_port and userstore_db_port.

9. Either download pre-built cartridge images from Apache Stratos website or create your own cartridges. Please refer Apache Stratos documentation for more information on creating cartridge images. For Amazon EC2, you could find pre-built PHP, MySQL and Tomcat cartridges published in Amazon EC2 AMI image repository.

10. If all stratos products are installed on the same node, update /etc/hosts file with a set of domain/host names mapping each product. These values should be updated on conf/setup.conf file. If they are installed on different nodes use actual hostnames.

    <ip-address> stratos.apache.org        # stratos domain
    <ip-address> mb.stratos.apache.org     # message broker hostname
    <ip-address> cc.stratos.apache.org     # cloud controller hostname
    <ip-address> sc.stratos.apache.org     # stratos controller hostname
    <ip-address> elb.stratos.apache.org    # elastic load balancer hostname
    <ip-address> agent.stratos.apache.org  # agent hostname

11. Update ./conf/setup.conf and configure parameters. 

12. Run setup.sh as root to install.

    sudo ./setup.sh -p "<product-list>"
    <product-list> could be defined as "cc sc elb agent" or any other combination according to the
    deployment configuration.
    Example:
    sudo ./setup.sh -p "all"

13. If you need to clean the setup run bellow command:
      sudo ./clean.sh -u <mysql-username> -p <mysql-password>


Friday, August 16, 2013

Review - A New Design for Distributed Systems : The Remote Memory Model

What problem does the paper attack? How does it relate to and improve upon previous work in its domain?

Existing distributed systems relay on clients which has its own memory used as their virtual memory. This paper discussed about, the conventional virtual memory systems in distributed environment and the limitations of them. The paper addresses the solution for those limitations and they introduced a new design called “The remote memory model”. Also it describes the impact of remote memory model architecture for design of distributed systems.  

In existing virtual memory system architecture, each program has a large, linear address space in which it places code and data. Most conventional virtual memory systems use demand paging technique to retrieve data from secondary storage in on demand manner when system needs to access data. Existing virtual memory architecture cannot be used in distributed systems which contain diskless clients. So this paper addresses “The remote memory model” as a solution for this problem in distributed systems. 

What are the key contributions of the paper?

The main contribution of this paper is to designing remote memory model for distributed systems. Basically the remote memory model can be divided into three parts. 

  • Several client machines 
  • Various server machines, one of more dedicated machines called remote memory servers, 
  • A communication channel

In this model client machines used communication channel to access memory on the remote memory server. This model supports for heterogeneous client machines. It keep this model as general as possible. To enhance this new model from conventional systems, authors consider about following properties as well. 

  • Additional memory 
  • Arbitrarily large storage capacity 
  • Data sharing 
  • Offloading file server
  • Remote memory semantics

Communication protocol is another contribution of this paper which support for remote memory model. This protocol provides reliability, architecture independence, and efficiency along with remote memory server. Also they introduced two communication protocol layers called Xinu Paging Protocol (XPP) layer, and the Negative Acknowledgement Fragmentation Protocol (NAFP) layer.

Comments

The name of this paper is “A New Design for Distributed Systems: The Remote Memory Model”. Reading the title of this paper, reader gets an idea that this paper relevant to distributed system category. But the content of this paper did not talk anything about distributed systems. Authors did not mention any literature background about distributed systems. They entirely talked about their design model.  
  
Authors did not clearly specify the virtual memory problem in distributed systems. They just mentioned there was a problem and the remote memory model is the solution for that. But they did not comparer and prove their remote memory model is suitable for existing problems in distributed systems. They only experimented there prototype with several client machines and different kind of operating systems and showed the results. But they did not compare those results with existing systems. Also they did not talked about the communication latency with the remote memory server, because existing systems did not have such latency. Future works of this remote memory model also not mention in this paper.

References
[1] D. Comer and J. Griffioen, “A new design for distributed systems: The remote memory model,” in Proceedings of the USENIX Summer Conference, pp. 127–135, Citeseer, 1990

Review - SwissQM: Next Generation Data Processing in Sensor Networks

What problem does the paper attack? How does it relate to and improve upon previous work in its domain?

Existing systems in Data processing in sensor networks are limited in two fundamental ways. Those limitations are lack of data independent and poor integration with the higher layers of the data processing chain. In this paper they presented new technique for data processing in sensor networks called SwissQM. It support adaptability, multiple user and applications, high quality turn-around, extensibility and optimizes used of support when comparing to other data processing techniques in sensor networks.

SwissQM borrowed many ideas mainly from TinyDB. TinyDB is the declarative database abstraction layer under TinyOS and it provides in network query processing and aggregation. But comparing SwissQM along with TinyDB, SwissQM is intended as the next generation query processing in sensor networks. It provides richer and more flexible functionality at the senor network level and powerful adaptable layer to the outside world. Also it provides data independent, query language independent and optimized perform in wide range when comparing with existing systems such as TinyDB, Giotto  a runtime environment for embedded systems. Rather than other systems, SwissQM is based on a special machine that runs optimized byte code rather than queries. Also it provides generic high-level declarative programming model and impose no data model. Comparing SwissQM with other data processing techniques, SwissQM used new technique to query processing. SwissQM translate the query for byte code. Therefore the rage of expressions supported by SwissQM is not limited. Also it supports multi-query on two layers. First it merges different user queries into virtual query and then it performs multi-programming in the query machine. 

What are the key contributions of the paper?

Going through this paper we can figure out SwissQM’s design considerations has become the key contributions of this paper. SwissQM has been designed to fulfill several key requirements which had not fulfilled by other data processing techniques in sensor networks. Those are 

  • Separation of sensors and external interface
  • Dynamic, multi-user, multi-programming environment 
  • Optimized use of sensors   
  • Extensibility

These novel design principles are helpful to SwissQM become the next generation data processing in sensor networks. They categorize SwissQM sensor network for four categories and shown the performances of this new design approach. SwissQM sensor networks consist with the sensor network gateway, the query machine, query machine byte code and query programs. 

Comments

Separation between gateway and sensor nodes and the implementation of a virtual machine at the sensor nodes, rather than the query processor are the main key design decisions in SwissQM. These two key designs gives the Turing-completeness, independent of query language used, independent of the user model and the extensibility of SwissQM. 

Rather than presenting the features, this paper mentioned few examples of SwissQM queries and explained those queries preciously. They used well known compiler technique called “templating” to translate virtual queries to SwissQM queries. Also they clearly illustrate init, delivery and reception which are the main three QM program sections by using different examples. 
Memory is the utmost crucial factor when considering the sensor networks. SwissQM has fifty nine instructions set and they showed the complete instruction set need 33kB flash memory and 3kB SRAM memory. This is a big advantage when comparing with TinyDB because it takes 65kB of flash memory and 3kB of SRAM memory. 

Message size also becomes the crucial factor in ad-hoc sensor networks. Different radio platforms provide different massage sizes. SwissQM has addressed the solution for this problem. In SwissQM they were opted message size for 36 bytes. But TinyDB uses 49 bytes. This is big step in ad-hoc sensor networking data processing. 

SwissQM has used three-tier architecture for visualizing the sensor network and permit multi-query optimization for more efficient use of sensor networks. It used query merging, sub expression matching and window processing optimization for multi-query optimization. By using this three-tier architecture the optimized virtual queries are transformed into network queries and those networks queries can be easily understood by the sensor nodes.

References
[1] R. Mueller, G. Alonso and D. Kossmann, "SwissQM: Next generation data processing in sensor networks," in Third Biennial Conference on Innovative Data Systems Research, Asilomar, CA, USA, January 7-10, 2007

Abstract - NoSQL Query Processing System for Wireless ad-hoc and Sensor Networks By Manula Thantriwatte & Chamath Keppetiyagama

Wireless Sensor Networks (WSN) are distributed systems typically composed of embedded devices, each equipped with a processing unit, a wireless communication interface, as well as sensors and/or actuators. Many applications have been implemented on sensor networks demonstrating the versatility of this technology, and some are already finding their way into the mainstream. Often, sensor nodes are tiny battery powered devices. Usually, these devices are capable of forming ad hoc wireless networks and communicate with each other.

Two main abstractions are available for developing applications for WSNs; message passing and SQL query interfaces. First exposes the network to the programmers and the second hides the complexity of the network with a database abstraction.  There are two popular query processing systems for wireless sensor network, namely TinyDB (for TinyOS) and TikiriDB (for Contiki). These query processing systems represent the sensors on the sensor network as a table. Users can insert queries at the base station, and it converts those queries into sensor node understandable format, and they are sent to sensor network to get the results.

Database abstractions currently used in WSN are based on Relational Database Management Systems (RDBMS). These relational models use ACID properties (atomicity, consistency, isolation, durability). TinyDB and TikiriDB use SQL queries.  However, a relational database model that guarantees ACID properties is not a good match for a wireless sensor network since consistent connectivity or the uninterrupted operation of the sensor nodes cannot be expected.

We noted that the NoSQL approach which does not rely on the ACID properties is a better match for a query processing systems for WASNs.  We developed a NoSQL based database abstraction on the Contiki operating system that is popular among the WSN community.

We designed NoSQL query syntaxes for querying sensor networks that are similar to the RedisDB NoSQL queries and we also adopted the RedisDB architecture for our implementation. We implemented the following NoSQL queries on Contiki.

  • Select Query: Appropriate keyword followed by relevant key
  • Join Query: Appropriate keyword followed by relevant key followed by valid set condition for key
  • Range Query: Appropriate keyword followed by relevant key followed by valid range condition
  • Ranking Data: Appropriate keyword followed by key and relevant member name
  • Get the key of members: Appropriate keyword followed by relevant key

We have implemented the Redis back-end in iterative manner. RedisDB supports different data structures such as Strings, Hashes, Lists, Sets and Sorted-sets. We prototyped the back-end using each of the above mentioned data structures and evaluated the performance.  Based on the experience gained through these prototypes, we decided to use Sorted-sets for the final implementation.

Sorted-set implementation of our NoSQL database abstraction works with two values called the key and the member. In this implementation we mapped sensing field of a query to a key and sensing values to members. According to these key and member values we can use following NoSQL queries in our database abstraction.

  • ZADD key score member: Add a member to a Sorted-set, or update its score if it already exists.
  • ZCARD key: Get the number of members in a Sorted-set.
  • ZCOUNT key min max: Count the members in a Sorted-set with scores within the given values.
  • ZINCRBY key increment member: Increment the score of a member in a Sorted-set. 
  • ZINTERSTORE destination numkeys key [key ...]: Intersect multiple Sorted-sets and store the resulting Sorted-set in a new key.

In performance analysis we analyzed the query execution time of TikiriDB database abstraction and our newly designed NoSQL database abstraction. We collected the execution times of queries by changing the sample period of both SQL and NoSQL queries. Then we observed that for a shorter time periods, both database abstractions show the same performances, but when the time period increases, performances of NoSQL database abstraction get better. The reason for that performance bottleneck in SQL database abstraction is the processing time of SQL queries. According to above results we can conclude that processing NoSQL queries are much more efficient than processing SQL queries in sensor networks. 

Also we measured the power required to process the NoSQL queries as well as the SQL queries. Our implementation consumes less power than the SQL based query processing system on Contiki. This is important since sensor nodes are usually energy constrained and an efficient query processing system helps to conserve energy.

References

Thursday, August 15, 2013

Apache Stratos

Introduction
Apache Stratos (Incubating) is a polyglot PaaS framework, providing developers a cloud-based environment for developing, testing, and running scalable applications, and IT providers high utilization rates, automated resource management, and platform-wide insight including monitoring and billing. [1]

Apache Stratos helps run Tomcat, PHP and MySQL apps as a service on all the major cloud infrastructures. It brings self-service management, elastic scaling, multi-tenant deployment, usage monitoring as well as further capabilities. In addition, it brings the ability to take any server software and make it into a Stratos Cartridge. This means any server can be run 'as-a-Service' alongside the other app containers. [1]

Layered Architecture
According to the diagram middle layer is the stratos foundation which consists of Elastic Load Balancer, Cloud Controller, Stratos Controller and CLI / WebUI Tool. On top of that there are some services such as Logging Service, Registry Service, Messaging Service and Billing Service. We can configure those services with the top layer. After you configure the foundation layer with this services you can use these services in the run time.

Cartridges
A Cartridge is a package of code or configuration that plugs into Stratos to offer a new PaaS Service. A Cartridge is also a Virtual Machine (VM) image plus configuration and it can operate in two modes that are namely single tenant and multi-tenant.

For more information see Cartridges.

References

Friday, July 5, 2013

Understanding & Using Git

What is git ?
Git is a distributed version control and source code management system designed and initially it was designed and developed by Linus Torvalds.

Every Git working directory is a full-fledged repository with complete history and full version tracking capabilities, not dependent on network access or a central server.

Git is free software distributed under the terms of the GNU General Public License version 2.

Getting a git repository
You can get a Git project using two main approaches. The first takes an existing project or directory and imports it into Git. The second clones an existing Git repository from another server.

Initializing a repository in an existing directory
If you’re starting to track an existing project in Git, you need to go to the project’s directory and type 
  • $ git init
This creates a new sub directory named .git that contains all of your necessary repository files to a Git repository skeleton. At this point, nothing in your project is tracked yet.

Cloning an existing repository
If you want to get a copy of an existing Git repository - for example, a project you’d like to contribute to the command you need is git clone. If you’re familiar with other VCS systems such as Subversion, you’ll notice that the command is clone and not checkout. This is an important distinction. Git receives a copy of nearly all data that the server has. Every version of every file for the history of the project is pulled down when you run git clone. In fact, if your server disk gets corrupted, you can use any of the clones on any client to set the server back to the state it was in when it was cloned.

You clone a repository with git clone [url]. For example, if you want to clone the Ruby Git library called Grit, you can do so like this:
  • $ git clone git://github.com/schacon/grit.git

The three states
Now, pay attention. This is the main thing to remember about Git if you want the rest of your learning process to go smoothly. Git has three main states that your files can reside in: committed, modified, and staged. Committed means that the data is safely stored in your local database. Modified means that you have changed the file but have not committed it to your database yet. Staged means that you have marked a modified file in its current version to go into your next commit snapshot.

This leads us to the three main sections of a Git project: the Git directory, the working directory, and the staging area.



The Git directory is where Git stores the metadata and object database for your project. This is the most important part of Git, and it is what is copied when you clone a repository from another computer.

The working directory is a single checkout of one version of the project. These files are pulled out of the compressed database in the Git directory and placed on disk for you to use or modify.

The staging area is a simple file, generally contained in your Git directory, that stores information about what will go into your next commit. It’s sometimes referred to as the index, but it’s becoming standard to refer to it as the staging area.

The staging index
Git internally holds a thing called the index, which is a snapshot of your project files. After you have just created an empty repository, the index will be empty. You must manually stage the files from your working tree to the  index using git add:
  • $ git add somefile.txt
git add works recursively, so you can also add whole folders:
  • $ git add somefolder

Committing
git commit takes the contents of the index and creates a new commit:
  • $ git commit somefile.txt -m "comment" 
Or if you want to commit all the changes at ones
  • $ git commit -a -m "comment"

Committing is a completely local operation, not related to sending something to a remote server. It just takes the contents of the index and keeps a snapshot of your project files as they were in the index:

View committing history
You can view the committing history by using
  • $ git log

Push all commits to the main repository
You can push all commits to the main repository by using
  • $ git push --all
This will push all the commit changes in all branches. If you want to push a particular branch you can use
  • $ git push -u origin <branch name>
This command only push the given branch to the main repository.

Pull others changes
You can get others changes to your local repository by using
  • $ git pull

Git branching, checkout & merge
If you init git or get the clone from the url you will get the master branch. As a best practice, it's better to have different branches for your changes.

As an example suppose if you want to do some platform changes in your code. To do that you can make a branch from the master branch in your local repository.
  • $ git branch platform
Then you can checkout that branch by using
  • $ git checkout platform
After that you can do the modification and commit the changes to that particular branch you created. Finally if you want to apply those changes to the master branch in your local repository, first you have to checkout the master branch. After that you can merge the changes by using following git command.
  • $ git merge platform
You can see what's happen on by using git status command.

Make patch using git
Suppose if you need to make a patch from your changes. The command is,
  • $ git format-patch -1 branch_name
In here -1 means, your patch will create using last commit changes. If you use -2 here, it makes two patches with last two commits.

Checkout a branch from the main repository
Suppose if you want to checkout the branch from the mail repository. To do that you can use the following command.
  • $ git checkout -b branch_name /remote/origin/branch_name

Get the specific file from the main repository
Suppose if you want to get the specific file from the main repository to your local repository. This will replace your file with the repository file. To do that you can use
  • $ git checkout -- filename

Stashing the changes
Often, when you’re been working on part of your project, things are in a messy state and you want to switch branches for a bit to work on something else. The problem is, you don’t want to do a commit of half-done work just so you can get back to this point later. To do that you can stash your work by using
  • $ git stash
Stashing takes the dirty state of your working directory - that is, your modified tracked files and staged changes - and saves it on a stack of unfinished changes that you can reapply at any time.

Then you can switch to another branch and do the changes what you want and make a commit. Suppose if you want to reapply the changes, first you have to switch that particular branch. After that you can view the stashing status by using
  • git stash list
This will give the list of stash you have done. For example, it'll give something like this.

stash@{0}: WIP on master: 049d078 added the index file
stash@{1}: WIP on master: c264051... Revert "added file_size"
stash@{2}: WIP on master: 21d80a5... added number to log

In this case, two stashes were done previously, so you have access to three different stashed works. You can reapply the one you just stashed by using the command
  • $ git stash apply
If you want to apply one of the older stashes, you can specify it by naming it, like
  • $ git stash apply stash@{2}
If you don’t specify a stash, Git assumes the most recent stash and tries to apply it.


Friday, June 14, 2013

Completely Uninstall MySQL from Ubuntu and Reinstall

To remove MySQL completely from your system just type in terminal

  • sudo apt-get remove --purge mysql-server mysql-client mysql-common 
  • sudo apt-get autoremove 
  • sudo apt-get autoclean

After that make sure to Removed /var/lib/mysql

  • sudo rm -r /var/lib/mysql

Then you have successfully remove mysql from your system. Finally you can install mysql again by typing following command in terminal

  • sudo apt-get install mysql-server

Monday, June 10, 2013

NoSQL Based Middleware for Sensor Data Management


Along with the rapid development of technology, sensors are more and more intelligent, autonomous, small and cheap. Different kinds of sensors are already being used in various domains such as medical, environmental, military, urban and industrial. These applications require different kinds of data such as temperature, humidity, pressure, GPS values. For managing these types of different sensor data and retrieving information, we need to have heterogeneous middleware system. We noted that the NoSQL approach which does not rely on the ACID properties is a better match for a database abstraction for sensor networks. Thus we proposed NoSQL based middleware for managing these heterogeneous distributed sensor data.

Introduction 
Sensor networking is an emerging technology and it has become increasingly attractive area for researchers. The applications for sensor networks are varied; typically those applications are involving tasks such as monitoring, tracking or controlling systems. Area monitoring, environmental monitoring, green house monitoring, landside detection, machine health monitoring and fleet monitoring are some of the areas for sensor networks [1].

Retrieving information from these different kinds of heterogeneous data is more challenging task. In order to retrieve data we have to use middleware platform [2, 3]. However the problem is; we have to deal with application dependent data to retrieve information. As a solution to these problems we proposed NoSQL based middleware for dealing with these application dependent data and retrieving information from heterogeneous sensor data, triggered from various applications. 

Dealing with large set of heterogeneous data, using a table based database systems, it needs lot of resources to store such a massive data and the operations are time consuming to retrieve information. With regards to NoSQL databases [4], it handle massive amount of data in much easier manner, and the performances wise it is very faster comparing with relational database management systems.

Problem and Motivation
Even though there are middlewares developed for sensor data management, all those are depending on relational aspect based on ACID properties [2]. However the current middleware systems in sensor data management faced scalability, mobility and heterogeneity problems. Therefore with NoSQL Middleware approach we can overcome those limitations, because NoSQL guarantees the scalability of data by retrieving information from heterogeneous data [5].

Literature Review
Existing sensor data management systems can be categorized as distributed sensor networks [2, 3, 6], centralized sensor networks [7, 8] and hybrid sensor networks [9, 10]. In distributed sensor networks, sensors are usually supposed to be homogeneous and they used same communication protocol [3]. This is a limitation for deploying large scale sensor network. In centralized sensor network systems, heterogeneity is not much explored. Also data retrieving queries are expensive and that is a limitation for the lifetime of the sensor network. In hybrid approach, we can use heterogeneous sensors. Due to that, scalability of the sensor data is very much higher than other sensor network categories.

For the time being there is a NoSQL query processing system for wireless ad-hoc and sensor networks [11]. It was proven that NoSQL based query processing system is better than other query processing systems such as TinyDB [2], TikiriDB [12], Cougar [13, 14] and Vector Programming database abstraction [15].

Proposed Methodology
Due to the scalability of data in hybrid sensor network approach, we have to introduce new kind of middleware for sensor data management. NoSQL (eg: - RedisDB [16], CouchDB [5], Cassandra [4], LevelDB) is the best data management system for highly scalable data. As discussed under problem and motivation sections; limited power resources, scalability, mobility and dynamic network topologies are the major challenges to design middleware for sensor data management. For overcome these problems, NoSQL based middleware is a better match for sensor data management.

Going through NoSQL based middleware for sensor data management we can maintain the high scalability of data to retrieve information from distributed heterogeneous sensor data, quick response to queries and retrieve information quickly, low energy consumption and we can use it within limited memory capacity.

Expected Outcome
Using NoSQL based middleware for sensor data management, we can achieve,
  • Quickly retrieving information from highly scalable heterogeneous sensor data
  • Low energy consumption
  • Low memory usage
  • Increase sensor network lifetime
References
[1] C. Weyer, V. Turau, A. Lagemann, and J. Nolte, “Programming wireless sensor networks in a selfstabilizing style,” in Sensor Technologies and Applications, 2009. SENSORCOMM’09. Third International Conference on, vol. 0, pp. 610–616, IEEE, 2009.

[2] S. R. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong, “TinyDB: an acquisitional query processing system for sensor networks,” ACM Transactions on Database Systems, vol. 30, pp. 122–173, Mar. 2005.

[3] P. Bonnet, J. Gehrke, and P. Seshadri, “Towards sensor database systems,” in Mobile Data Management, pp. 3–14, Springer, 2001.

[4] A. Lakshman and P. Malik, “Cassandra: a decentralized structured storage system,” ACM SIGOPS Operating Systems Review, vol. 44, no. 2, pp. 35–40, 2010.

[5] M. Seeger and S. Ultra-Large-Sites, “Key-value stores: a practical overview,” Computer Science and Media, Stuttgart, 2009.

[6] M. Chen, “Energy-efficient differentiated directed diffusion (EDDD) in wireless sensor networks,” Computer Communications, vol. 29, pp. 231–245, Jan. 2006.

[7] D. Abadi, D. Carney, U. Cetintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik, “Aurora: a new model and architecture for data stream management,” The VLDB Journal, vol. 12, no. 2, pp. 120–139, 2003.

[8] D. P. Arvind, A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, I. Nishizawa, J. Rosenstein, and J. Widom, “Stream: The stanford stream data manager,” IEEE Data Engineering Bulletin, vol. 26, pp. 19–26, 2003.

[9] S. Madden and M. Franklin, “Fjording the stream: An architecture for queries over streaming sensor data,” in Data Engineering, 2002. Proceedings. 18th International Conference on, pp. 555–566, IEEE, 2002.

[10] D. Abadi, W. Lindner, S. Madden, and J. Schuler, “An integration framework for sensor networks and data stream management systems,” in Proceedings of the Thirtieth international conference on Very large data bases-Volume 30, pp. 1361–1364, VLDB Endowment, 2004.

[11] T. A. M. C. Thantriwatte and C. I. Keppetiyagama, “Nosql query processing system for wireless ad-hoc and sensor networks,” in Proceedings of the International Conference on Advances in ICT for Emerging Regions (ICTer), 2011, ICTer 2011, pp. 78–82, IEEE, 2011.

[12] N. M. Laxaman, M. D. J. S. Goonathillake, and K. D. Zoysa, “TikiriDB : Shared Wireless Sensor Network Database for Multi-User Data Access,” CSSL 2010.

[13] Y. Yao, “The cougar approach to in-network query processing in sensor networks,” ACM SIGMOD Record, vol. 31, p. 9, Sept. 2002.

[14] W. F. Fung, U. Hall, D. Sun, U. Hall, J. Gehrke, and U. Hall, “COUGAR : The Network is the Database,” System, no. July 2001, pp. 14853–14853, 2002.

[15] T. Sookoor and K. Whitehouse, “Vector Programming Abstraction for Sensor Networks,” 2010.

[16] Citrusbyte, “Redis.” Website, August 10 2012.