Thursday, October 22, 2015

Datastage -Functions in Transformer


1.  AlNum(%string%)
Allnum(“12345”) --- output-1
Allnum(“abcd”) --- output-1
Allnum(“ab125”) --- output-1
Allnum(“@a12345”) --- output-0
Allnum(“@a1 2345”) --- output-0
Except Special Symbols (Letters & Numbers) it will give output as “1”
2. Alpha(%string%)
Alpha (“abcd”) --- output-1
Alpha (“12345”) --- output-0
Alpha (“ab125”) --- output-0
Alpha (“@a12345”) --- output-0
Alpha (“@a1 2345”) --- output-0
3. Compactwhitespace(%string%)
     It reduces or replaces multiple spaces into single space.
Ex: CompactWhiteSpace("sud hee                  r") 
            Out Put: sud hee r
4. Space(length):

     Inserts no.of white spaces given in lengthy parameters.
        Ex:-Space(6):’Brahma’
       Output: Brahma

5 . Len(string):
Returns length of string in characters
     Ex:- Len(space(6):’Brahma’)
     Output---12
     Ex:- Len(“Brahma”)
      Output—6
6. Compare(String1,String2,[Justification]):       //case sensitive
      Compares String1 & String2,if String1 is greater than String2 it will result 1
            If String1 is less than String2,then it will result 1.
            If both the strings are equal then it will result 0.
            To compare String1 with String2, we give Justification:L
           To compare String2 with String1,we give Justification:R
     Ex:       Compare(“abcd”,”abc”,”L”)---output-1
                  Compare(“abcd”,”abc”,”R”)---output-0
                  Compare(“abc”,”Abc”)---output-1     Default Justification-L
                  Compare(“abc”,”abcd”)---output  -1
7. CompareNoCase(String1,String2):
             Same as Compare,but diff is this function is NOT case sensitive.
          In this function no argument called Justification.
8. CompareNum(String1,String2,length):
           Compares two strings up to the given length.
           Ex:-     CompareNum(“a”,”abc”,1)     output---0
                       CompareNum(“a”,”abcde”,5) output--    -1
           CompareNum(“abcde”,”a”,5) output---1
           CompareNum(“abcde”,”Abcde”,1)    output---1
           CompareNum(“Abcde”,”abcde”,1)    output--   -1
9. CompareNumNoCase(String1,String2,length):
          Same as CompareNum, diff is this function is NOT case sensitive.
          Ex:-     CompareNumNoCase(“Abcde”,”abcde”,5)   output---0
         CompareNumNoCase(“abxye”,”abcde”,5)    output---1
         CompareNumNoCase(“abcde”,”abxde”,5)    output--  -1
10. Upcase(String):
           Change all lowercase letters in a string to uppercase.
           Ex:-     Upcase(“brahma”)       output—BRAHMA
           Upcase(“braHma”)      output----BRAHMA
11. DQuote(String):
          Enclose a string in double quotation marks.
         Ex:-     DQuote(Upcase(“brahma”))   output---“BRAHMA”
        DQuote(‘brahma’)                  output---“brahma”
12. Field(String,delimiter,occurrence,number):
          Returns the substring before delimiter based on occurrence we are given.
            Ex:-     Field(“br_ah_ma_na”,”_”,2)   output---ah
         Here ah is substring before the second occurrence of delimiter ‘_’.
Field(“br_ah_ma_na”,”_”,3)   output---ma
         Here after ma _ is third occurrence.
Field(“br_ah_ma_na”,”_”,1,4)            output---br_ah_ma_na
Field(“br_ah_ma_na”,”_”,1,2)            output---br_ah
Field(“br_ah_ma_na”,”_”,1,3)            output---br_ah_ma
Field(“br_ah_ma_na_a”,”_”,1,5)        output---br_ah_ma_na_a
 Note:-
          Field(“brahmananda”,”a”,2)   output—hm
   Here ‘a’ is delimiter
         Field(“brbhmananda”,”a”,2)   output—n
         Field(“brhhmananda”,”a”,1)   output—brhhm
          Field(“brahmananda”,”a”,1,3)  output—brahman

13. Index(String,substring,occurrence):
        Returns starting character position of substring.
        Ex:-     Index(“brahma”,”a”,1)            output---3
        Index(“brahma”,”a”,2)            output---6
        Index(“brahmananda”,”a”,3)  output---8
        Index(“brahmananda”,”na”,1)  output---7
14. Convert(fromlist,tolist,Expression)
      Converts specified character in a string(given in expression arg) to designated replacement character
      Ex:-  convert(“a”,”y”,”brahma”)
   o/p:byhmy
             convert(“ah”,”y”,”brahma”)
   o/p:brymy
      This function performs character replacement, but not word.
      Ex:  convert(“brahma”,”msrmad”,”brahma”)
15. Count(String,Substring):
      Count number of times a Substring occurs in a String.
      Ex:Count(“brahma”,”a”)
            o/p:2
           Count(“brahmabaabaaba”,”ab”)
            o/p:3
          Count(“brahmabaabaaba”,”ba”)
            o/p:3
16. DCount(String,Delimiter):
      Count number of delimited fields in a string.
     Ex:  DCount(“br-ah-ma-na”,”_”)-------o/p:4
             DCount(“br,ah,ma,na”,”,”)-------o/p:5
             DCount(“br,ah-ma,na”,”,”)-------o/p:4
    ah-ma, treats as one field
17. Downcase(String):
       Change all uppercase letters in a String to lowercase
     Ex:  Downcase(“BRAHMA”)------o/p:brahma
             Downcase(“brHMA”)------o/p:brahma
             Downcase(“BRAhma”)------o/p:brahma
18. Left(String,length)
        Returns leftmost ‘n’ characters of the string, where n is length.
      
       Ex:  Left(“brahmananda”,4) o/p:brah
               Left(“bra__hmananda”,6) o/p:bra_hm
19. Right(String,Length)
      Returns rightmost ‘n’ characters of the string
    
      Ex:   Right(“brahmananda”,4) o/p:anda
20. Num(String)
     Returns 1 if string can be converted to a number.
     
     Ex:    Num(“brah”)    o/p=0
              Num(“369”)     o/p=1
21. PadString(String,PadString,Padlength)
       Returns the string padded with the optional pad character and padlength is the number of times it add to the original string
    
      Ex:    PadString(“brahma”,”+”,5)    o/p:brahma+++++
                PadString(“brahma”,”reddy”,5)   o/p:brahmarrrrr
22. Str(String,repeats)
       Repeats the input string no.of times given in repeats
    
      Ex:  Str(“mbnr”,2)    o/pmbnrmbnr
23. StripWhiteSpace(String)
       Returns the string after stripping (removing) all white space from it.
     
     Ex:  StripWhiteSpace(“br a h ma”)   o/p:brahma

Wednesday, October 14, 2015

Top 20 Datastage Interview Questions

1. What is the flow of load loading data into fact & dimensional tables?
A) Fact table - Table with Collection of Foreign Keys corresponding to the Primary
Keys in Dimensional table. Consists of fields with numeric values.
Dimension table - Table with Unique Primary Key.
Load - Data should be first loaded into dimensional table. Based on the primary key
values in dimensional table, the data should be loaded into Fact table.
2. What is the default cache size? How do you change the cache size if needed?
A. Default cache size is 256 MB. We can increase it by going into Datastage
Administrator and selecting the Tunable Tab and specify the cache size over there.
3. What are types of Hashed File?
A) Hashed File is classified broadly into 2 types.
a) Static - Sub divided into 17 types based on Primary Key Pattern.
b) Dynamic - sub divided into 2 types
i) Generic ii) Specific.
Dynamic files do not perform as well as a well, designed static file, but do perform better
than a badly designed one. When creating a dynamic file you can specify the following
Although all of these have default values)
By Default Hashed file is "Dynamic - Type Random 30 D"
4. What does a Config File in parallel extender consist of?
A) Config file consists of the following.
a) Number of Processes or Nodes.
b) Actual Disk Storage Location.
5. What is Modulus and Splitting in Dynamic Hashed File?
A. In a Hashed File, the size of the file keeps changing randomly.
If the size of the file increases it is called as "Modulus".
If the size of the file decreases it is called as "Splitting".
6. What are Stage Variables, Derivations and Constants?
A. Stage Variable - An intermediate processing variable that retains value during read
and doesn’t pass the value into target column.
Derivation - Expression that specifies value to be passed on to the target column.
Constant - Conditions that are either true or false that specifies flow of data with a link.
7. Types of views in Datastage Director?
There are 3 types of views in Datastage Director
a) Job View - Dates of Jobs Compiled.
b) Log View - Status of Job last run
c) Status View - Warning Messages, Event Messages, Program Generated Messages.
8. Types of Parallel Processing?
A) Parallel Processing is broadly classified into 2 types.
a) SMP - Symmetrical Multi Processing.
b) MPP - Massive Parallel Processing.
9. Orchestrate Vs Datastage Parallel Extender?
A) Orchestrate itself is an ETL tool with extensive parallel processing capabilities and
running on UNIX platform. Datastage used Orchestrate with Datastage XE (Beta version
of 6.0) to incorporate the parallel processing capabilities. Now Datastage has purchased
Orchestrate and integrated it with Datastage XE and released a new version Datastage 6.0
i.e Parallel Extender.
10. Importance of Surrogate Key in Data warehousing?
A) Surrogate Key is a Primary Key for a Dimension table. Most importance of using it is
it is independent of underlying database. i.e. Surrogate Key is not affected by the changes
going on with a database.
11. How to run a Shell Script within the scope of a Data stage job?
A) By using "ExcecSH" command at Before/After job properties.
12. How to handle Date conversions in Datastage? Convert a mm/dd/yyyy format to
yyyy-dd-mm?
A) We use a) "Iconv" function - Internal Conversion.
b) "Oconv" function - External Conversion.
Function to convert mm/dd/yyyy format to yyyy-dd-mm is
Oconv(Iconv(Filedname,"D/MDY[2,2,4]"),"D-MDY[2,2,4]")
13 How do you execute datastage job from command line prompt?
A) Using "dsjob" command as follows.
dsjob -run -jobstatus projectname jobname
14. Functionality of Link Partitioner and Link Collector?
A)Link Partitioner: It actually splits data into various partitions or data flows using
various partition methods.
Link Collector: It collects the data coming from partitions, merges it into a single data
flow and loads to target.
15. Types of Dimensional Modeling?
A) Dimensional modeling is again sub divided into 2 types.
a) Star Schema - Simple & Much Faster. Denormalized form.
b) Snowflake Schema - Complex with more Granularity. More normalized form.
16. Differentiate Primary Key and Partition Key?
A)Primary Key is a combination of unique and not null. It can be a collection of key values
called as composite primary key. Partition Key is a just a part of Primary Key. There are
several methods of partition like Hash, DB2, and Random etc. While using Hash partition
we specify the Partition Key.
17. Differentiate Database data and Data warehouse data?
A) Data in a Database is
a) Detailed or Transactional
b) Both Readable and Writable.
c) Current.
18. Containers Usage and Types?
A)Container is a collection of stages used for the purpose of Reusability.
There are 2 types of Containers.
a) Local Container: Job Specific
b) Shared Container: Used in any job within a project.
19. Compare and Contrast ODBC and Plug-In stages?
A)ODBC: a) Poor Performance.
b) Can be used for Variety of Databases.
c) Can handle Stored Procedures.
Plug-In: a) Good Performance.
b) Database specific. (Only one database)
c) Cannot handle Stored Procedures.
20. Dimension Modelling types along with their significance
A)Data Modelling is Broadly classified into 2 types.
a) E-R Diagrams (Entity - Relatioships).
b) Dimensional Modelling.

Tuesday, October 13, 2015

How to Start and Stop the Datastage Engine

Here are the instructions to Stop Information Server 8.x, verify all the processes are stopped and then restart Information Server (see instructions below for each step).

We will stop in this order:

  •       Stop DataStage
  •       Stop Node Agents
  •       Stop WebSphere

1.     Log into your Unix/Linux machine as root (you can run first part as dsadm). 
2.     Make sure that no one is using DataStage. 
3.     Check if there are any processes left by executing the following commands:

  • ps -ef | grep phantom - This would show processes from jobs
  • ps -ef | grep osh - This would show processes for parallel jobs
  • ps -ef | grep dsapi - This would show processes for client connection
  • ps -ef | grep dscs - This would show processes for client connection

These commands shouldn't return any results. If they return results, it means that there are still some processes. You need to wait for the users to stop their processes or kill the processes (using kill -9 PID) if you confirm that these are not longer valid connections .

4.     Check if DataStage RPC Daemon (dsrpc) is running by executing:

  • netstat -a | grep dsrpc

If dsrpc is running then you should see one line with the status LISTEN. If you get more lines it means that there are still some connections and you need to wait. If you see that dsrpc or other connections are in status CLOSE_WAIT then you also need to wait until these disappear (you can also reboot the machine)

5.     Stop DataStage (you have to run this from $DSHOME)

  • . ./dsenv
  • bin/uv -admin -stop

6.     Check to see if there are any shared memory segments left:

  • ipcs -m | grep ade

Note: If you are not running with the default itag of "ade" use that instead of "ade". For example if you have two DataStage instances one would be "ade" and the second would have another tag, say "801" or "efg".
If you get a result, take note of the ID and remove it (you will need root access for this part)
ipcrm -m ID

7.     Then check dsrpc again using


  • netstat -a | grep dsrpc

You shouldn't get any results. If you get any and/or the dsrpc is in CLOSE_WAIT then you need to wait until this it disappears (you can also reboot the machine)

8.     Stop the ASB Agent (you need to be root here)


  • Go to <InstallPath>/IBM/InformationServer/ASBNode/bin
  • Run ./NodeAgents.sh stop

Type 'yes' if you receive the following message: rm: remove write-protected regular file Agent.pid'?

9.     Check the agents have stopped:

  • ps -ef | grep java

To see full command in Solaris:

  • /usr/ucb/ps -auxww | grep java


If still see the Logging Agent or ASB Agent then you need to kill these. e.g. kill -9 PID

10.  Stop the WebSphere Server (you need to be root here)

  • Go to <InstallPath>/IBM/WebSphere/AppServer/bin
  • Run ./stopServer.sh server1

* if that does not work, try ./bin/stopServer.sh server1 -username wasadmin -password wasadmin
Now check that the WebSphere java processes is gone:

  • ps -ef | grep java

To see full command in Solaris:

  • /usr/ucb/ps -auxww | grep java

remove it if it exists with: kill PID (no -9 in the kill command)
Now, you can restart, in this order:


  •  Start WebSphere Server
  •   Start ASB Agent
  •  Start DataStage

1.     Start WebSphere Server as root

  • Go to <InstallPath>/IBM/WebSphere/AppServer/bin
  • Run ./startServer.sh server1

Wait until you get back a line saying: Server server1 open for e-business;

2.     Start ASB Agent as root

  • Go to <InstallPath>/IBM/InformationServer/ASBNode/bin
  • Run ./NodeAgents.sh start

3.     Start DataStage (as dsadm)


  • Go to $DSHOME
  • Run bin/uv -admin -start

tMap vs tJoin -Talend

  tMap is frequently used component for joins and lookup purpose, it is also use for verity of operations and transformations, whereas tJoin...