Bigdata sample project

I have created sample bigdata project and kept it in Github

https://github.com/gobis/BigDataProject


I'll be updating github link in this post. 

Reversing singly Linked List - Explanation

I have searched some sites to know how they are reversing linked list , yeah I found so many sites but none of them are explained . So , I conclude it  to do in my blog



Below are the working piece of code 

 #include <iostream.h>  
 #include <conio.h>  
 class Node  
 {  
  public:  
    int data ;  
    Node * next ;   
    Node(int in){data = in; }   // public param cons  
    };  
 Node* Reverse(Node * head )  
  {  
      Node * temp = NULL ;  
      Node * np = NULL;  
     while (head)  
      {  
       temp = head->next;  
       head->next = np;  
       np = head ;  
       head = temp;     
      }  
      return np ;  
  }  
  void Display(Node * head )  
   {  
    while (head)  
    {  
     printf("%d \t" , head->data);  
     head = head-> next ;    
    }  
   }  
 main()  
 {  
   Node * obj1= new Node(10) ;  
   Node * obj2= new Node(20) ;  
   Node * obj3= new Node(30) ;  
   Node * obj4= new Node(40) ;  
   obj1->next = obj2 ;  
   obj2->next = obj3 ;  
   obj3->next = obj4 ;  
   printf("Before reversing ");    
   Display(obj1);    
   Node * head = Reverse(obj1);  
   printf("\n After reversing ");    
   Display(head);  
   getch();    
 }  

Lets see the explanation of Important lines 
  before getting in to that , you need two variable
1. temp -> used to hold next element pointer and given back the address to head
2. np    -> Assign it to head->next

lets discuss the code line by line

step1 :        temp = head->next;   
 take the information of next , so that we can play with head->next

step2:        head->next = np ;
 yeah , In the next line we are changing head->next  ( anyway we are having this info intemp   for later use)
initially np  is null , but from the second iteration it has the info of prev element

step3:       np = head ; 
yeh , this step does the same what we want in step 2

step4:    head = temp ;

finally head should move to next element . fortunately this info is stored in temp 


thats all dudes , np will become your new head pointer .

 It may not be the great explanation , but it will be used to the beginners .

Changing datatype in Spark sql column

here is the screen shot to change the datatype of the spark column

AES 128 Encryption : padding bytes issue

Recently I have encountered the padding issue with AES 128 encryption.

Mobile app is connecting to the gateway (which is in C++)  and communicating with each other.We have planned to encrypt the data. in C++ they have standard way of doing encryption and Android also have Cipher class to do encryption.

the problem here is ,
     Gateway code, giving "00" as padding bytes.


In android, we have different variant that provides various padding bytes

 Cipher cipher = Cipher.getInstance("AES/ECB/PKCS7Padding", "BC");

 This will give padding bytes as 0F  

 Cipher cipher = Cipher.getInstance("AES/ECB/ZeroBytePadding", "BC");

 This will give padding bytes as 00  

based on the padding bytes, you can choose the variant.

Understanding of virtual function

It is a function or method can be overridden within an inheriting class by a function with the same signature ,In other words , the purpose of virtual function is , to allow customization of derived class implementation

   “Where there is a casting there is a virtual”

Base * ptr = new Base() ; //  No need of virtual

Derived * ptr = new Derived();  // No need of virtual

Base * ptr = new Derived(); // need virtual to call correct function

Key benefits of Virtual function :

     It is allowed us to do function over riding

       

    Class Base { virtual void fn1() ; };

    Class derived: public base {void fn1() ;}

   Base* ptr = new derived() ;

    Ptr->fn1() ; this calls the derived class function fn1()

Above class become ptr->_vptr->fn1() ;

Now , consider if fn1() is not declared as virtual , then   the above call would call the Base class function .

Best part of this concepts is , you can have an array ,type of  Base* . and you can append derived classes object without worrying about function call .

 It is avoiding memory leak

  

   lets look at the below code

  

    Class Base { void ~Base() ; };

    Class derived: public base {void ~derived() ;}

   

    Base* ptr = new derived() ;

    delete ptr ;

   oops , there is a memory leak , above call only call base class destructor  .

  

wanna avoid memory leak

  Class Base { virtual void ~Base() ; };

Now , it will use vptr , and it will call the derived class destructor and subsequently it will call base class destructor

Disadvantages or Cost of virtual function

1.      If a class contains virtual function , then vtable is created , which means additional memory space required for that

2.      virtual function are mapped at runtime through vtable . hence calling virtual function is slow compared to normal function

3. vptr (virtual pointer which points to vtable) is created for every object .Additional 4 byte required to store vptr

pictorial representation of virtual function  

At last ,  If a function declared as virtual in base class , then the function in all derived class are virtual only . No need to mention it explicitly using “virtual” keyword

Some FAQ:

1. What is the size of an empty class

Ans : 1 byte , because compiler provides minimum 1 byte to the object

2. what is the size of the class  class A { virtual void fn1();};

      Ans : 4 byte ,  now vptr (virtual pointer comes in to the picture)

      

3. What is pure virtual function

 Ans : A virtual function doesn’t have body

         Virtual void fn1() = 0;

4. Can we create object of the class which has pure virtual function

Ans : No , Compiler will check the body of the function while creating object of the class , since it doesn’t have the body we can’t create the object and compiler will throw an error at the time of compilation

Synchronous AsyncTask in Android

In Android , there are some APIs (specially network related ) can called from another thread. and there will be some situation you will be encountered that , the main thread needs result of the Asynctask thread .

well , it is quite easy in android

you have to use below piece of code

             new TimerTask().execute().get();


timertask is extends from AsyncTask .

ie Thread1 will wait for thread2's result .

JNI step by step

1. Add native method in your activity class 
    for instance : public native String display(boolean status);

2. open command prompt and go to bin/classes
    Example :~/workspace/test/bin/classes$ javah -jni com.example.test.MainActivity
    Abstract :~/workspace/test/bin/classes$ javah -jni <class name where u have added your native API> 

    as a result , a header file will be created in   "test/bin/classes/" 
3. Copy that header file and paste it in to jni folder (create jni folder in the project )

4. Using signature of the header , create the cpp file and have definition for that method

5. Create Android.mk file (add the cpp file and module name inside it)

6. go to the project 
    Example:~/workspace/test$ "ndk-build.cmd"
    result : this will create lib (under the libs folder)

7. now, you can call the native method from the application

Double Pointer (Pointer to Pointer) some input

A double pointer has two basic meanings.
        One is of a pointer to a pointer, where changing the value of double pointer will result in the original pointer being changed.
        Another is that of a two-dimensional array, such as a matrix

Example for Two dimensional array
int main( int argc, char **argv (or) char* argv[] )
{
// print the first string
printf("%s\n", argv[0]);

return 0;
}

The other example where the double pointer is a pointer to a pointer, lets say you have a function foo() that allocates memory for a pointer that was declared in function main()

void foo( char ** ptr)
{
*ptr = malloc(255); // allocate some memory
strcpy( *ptr, "Hello World");
}

int main()
{
char *ptr = 0;
// call function with a pointer to pointer
foo( &ptr );
printf("%s\n", ptr);
// free up the memory
free(ptr);

return 0;
}

Creating 2D int array
//Create your pointer
int **ptr;
//Assign first dimension
ptr = new int*[5];
//Assign second dimension
for(int i = 0; i < 5; i++)
ptr[i] = new int[5];
Deleting 2D array
for (int x = 0; x < nCol; ++x)
delete [] ptr[x];
delete [] ptr;

How delete[] knows how many items to be destroyed ??
 When an array is created with new[], the size of array is stored in the metadata at the memory block. delete[] makes use of that information.

New[] operator requires default constructor  , for ex
   student* kar = new student[10](34,"");  // its completely wrong
  student * kar = new student[10];   // Correct

new & delete are based on malloc & free. Malloc additionally allocate a header storing the size allocated, so free() knows the size to free. new[] not only keeps the malloc header, but also has an extra header storing the size of the array/vector, so delete[] knows how many objects are in the array/vector, and thus knows how many times to call the destructor.

Allocation Using	Deallocation using	Result	Remarks
new	delete	Ok
new[]	delete[]	Ok
new	delete[]	The vector delete will try to delete more objects depending on the random value it gets from its object-count location and will obviously result in heap corruption.	Compiler will not track the Pointer, Its developer responsibility to call the respective delete operator (ie delete or delete[])
new[]	delete	Memory leak	It will delete only first item in the array

android (Framework) bootup sequence

The System Server in Android
In this post I will add some more detail on the system server in Android. The system server is the core of the Android system
and as described in the boot sequence post it is started as soon as Dalvik is initialized and running.
The other system services will be running in the context of the System Server process.
We will start by looking at the code that runs when the System Server starts.
 This code is found in the file frameworks/base/services/java/com/android/server/SystemServer.java (in the open source project tree)
 and we will start this discussion from the main entry point.

 /**
     * This method is called from Zygote to initialize the system. This will cause the native
     * services (SurfaceFlinger, AudioFlinger, etc..) to be started. After that it will call back
     * up into init2() to start the Android services.
     */
    native public static void init1(String[] args);

    public static void main(String[] args) {
        // The system server has to run all of the time, so it needs to be
        // as efficient as possible with its memory usage.
        VMRuntime.getRuntime().setTargetHeapUtilization(0.8f);
       
        System.loadLibrary("android_servers");
        init1(args);
    }

    public static final void init2() {
        Log.i(TAG, "Entered the Android system server!");
        Thread thr = new ServerThread();
        thr.setName("android.server.ServerThread");
        thr.start();
    }

The first thing that happens is that the server will load a native library called android_servers that provides interfaces to native functionality.
Source files for this lib are placed in frameworks/base/services/jni/.
 Then the native init method that will setup native services is called, init1(args), and executed.
 The name of the function that implements this is system_init() and the it resides in frameworks/base/cmds/system_server/library/system_init.cpp.
 After setting up the native services there is a callback:


runtime->callStatic("com/android/server/SystemServer", "init2");

to init2() above to create the server thread. This thread will start the remaining services in the system according to the necessary start order.
A snippet of the initial sequence gives:

// Critical services...
        try {
            Log.i(TAG, "Starting Power Manager.");
            power = new PowerManagerService();
            ServiceManager.addService(Context.POWER_SERVICE, power);

            Log.i(TAG, "Starting Activity Manager.");
            context = ActivityManagerService.main(factoryTest);

            Log.i(TAG, "Starting telephony registry");
            ServiceManager.addService("telephony.registry", new TelephonyRegistry(context));

            AttributeCache.init(context);

            Log.i(TAG, "Starting Package Manager.");
            pm = PackageManagerService.main(context,
                    factoryTest != SystemServer.FACTORY_TEST_OFF);

            ActivityManagerService.setSystemProcess();

            mContentResolver = context.getContentResolver();

            Log.i(TAG, "Starting Content Manager.");
            ContentService.main(context,
                    factoryTest == SystemServer.FACTORY_TEST_LOW_LEVEL);

            Log.i(TAG, "Starting System Content Providers.");
            ActivityManagerService.installSystemProviders();

            Log.i(TAG, "Starting Battery Service.");
            BatteryService battery = new BatteryService(context);
            ServiceManager.addService("battery", battery);

            Log.i(TAG, "Starting Hardware Service.");
            hardware = new HardwareService(context);
            ServiceManager.addService("hardware", hardware);

            // only initialize the power service after we have started the
            // hardware service, content providers and the battery service.
            power.init(context, hardware, ActivityManagerService.getDefault(), battery);

            Log.i(TAG, "Starting Alarm Manager.");
            AlarmManagerService alarm = new AlarmManagerService(context);
            ServiceManager.addService(Context.ALARM_SERVICE, alarm);

...


We see that the power manager is started first, followed by the activity manager and the other services.
There are a lot more services started after these initial and if you are interested take look in the SystemServer.java file.
Each service is running in a separate Dalvik thread in the SystemServer process.
To give some info on the components making up the system server we may have look at it using the DDMS tool:

We see that the main Android services such as the activity manager, package manager, alarm manager etc.
are running in their separate threads but as parts of the system server process.