Assertions in TestNG

What is Assertion?
  •  Assertion determines the state of the application whether it is the same what we are expecting or not.
  •  While using Selenium for automated testing of web applications, we need to add validations in our tests to report them as pass or fail.
  •  Assertions give you a way (other than If-Else blocks) to test conditions.
There are two types of Assertions in TestNG: Hard Assertion(Assert) and Soft Assertion(Verify

Hard Assertions – It is the default assert mechanism built into TestNG’s “org.testng.Assert” package. We use it when a test has to stop immediately after the assertion fails.
Hard Assertion is an Assertion that throws the AssertException when the test case is failed. In the case of Hard Assertion, you can handle the error by using a catch block like a java exception. A Hard Assertion contains the following methods:
  • assertEquals
  • assertNotEquals
  • assertTrue
  • assertFalse
  • assertNull
  • assertNotNull  
Example: In below example we will understand Hard Assertion, in below test class we have two test methods and testCase1 test method is getting failed at 3rd line. Once exception occurs remaining code will not be executed in testCase1. However testCase2 will be executed successfully. Look at the output
import org.testng.Assert;
import org.testng.annotations.Test;

public class HardAssertionDemo {
 
 @Test
 public void testCase1() {
  System.out.println("==================================");
  System.out.println("Test1 Start");
  Assert.assertTrue(false);
  System.out.println("Test1 End");
  System.out.println("==================================");
 }
 
 @Test
 public void testCase2() {
  System.out.println("==================================");
  System.out.println("Test2 Start");
  Assert.assertTrue(true);
  System.out.println("Test2 End");
  System.out.println("==================================");
 }

}

Output:
==================================

Test1 Start

==================================

Test2 Start

Test2 End

==================================

PASSED: testCase2

FAILED: testCase1

java.lang.AssertionError: expected [true] but found [false]

 at org.testng.Assert.fail(Assert.java:96)

The disadvantage of Hard Assert – It marks method as fail if assert condition gets failed and the remaining statements inside the method will be aborted.

And sometimes we want to execute the whole script even if the assertion fails. This is not possible in Hard Assertion. To overcome this problem, we need to use a soft assertion in testng. 

Soft Assertions (Verify):
  • It is a custom assert mechanism supported by TestNG’s “org.testng.asserts.Softassert” package. We use it when a test has to continue execution even after an assertion fails in the sequence.
  • Soft Assert does not throw an exception when an assert fails and would continue with the next step after the assert statement.
If there is any exception and you want to throw it then you need to use assertAll() method as a last statement in the @Test and test suite again continue with next @Test as it is.

Example:
import org.testng.Assert;
import org.testng.annotations.Test;
import org.testng.asserts.SoftAssert;

public class SoftAssertionDemo {

 SoftAssert obj= new SoftAssert();

 SoftAssert obj1= new SoftAssert();


 @Test
 public void testCase1() {
  System.out.println("==================================");
  System.out.println("Test1 Start");
  obj.assertTrue(false);
  obj.assertEquals("Hi", "Hi");
  System.out.println("Test1 End");
  System.out.println("==================================");
  obj.assertAll();

 }
 @Test
 public void testCase2() {
  System.out.println("==================================");
  System.out.println("Test2 Start");
  obj1.assertTrue(true);
  obj1.assertEquals("HELLO", "HELLO");
  System.out.println("Test2 End");
  System.out.println("==================================");
  obj1.assertAll();

 }

}

Output:
==================================

Test1 Start

Test1 End

==================================

==================================

Test2 Start

Test2 End

==================================

PASSED: testCase2

FAILED: testCase1

java.lang.AssertionError: The following asserts failed:

 expected [true] but found [false]

Please refer below YouTube video on Assertions:

Listeners in TestNG

What is Listener?
As the name suggests Listeners "listen" to the event defined in the selenium script and behave accordingly. Listener is defined as interface that modifies the default TestNG's behavior.

Purpose of Listeners:
Listener is defined as interface that modifies the default TestNG's behavior. As the name suggests Listeners "listen" to the event defined in the selenium script and behave accordingly. It is used in selenium by implementing Listeners Interface. It allows customizing TestNG reports or logs. There are many types of TestNG listeners available.

There are two types of listeners:
  • WebDriver Listeners – events triggered by web driver.
  • TestNG Listeners - events triggered by TestNG.
TestNG Listeners:
  • IAnnotationTransformer ,
  • IAnnotationTransformer2 ,
  • IConfigurable ,
  • IConfigurationListener ,
  • IExecutionListener,
  • IHookable ,
  • IInvokedMethodListener ,
  • IInvokedMethodListener2 ,
  • IMethodInterceptor ,
  • IReporter,
  • ISuiteListener,
  • ITestListener
In this post, we will implement the ITestListener.

ITestListener has following methods
  • OnStart- OnStart method is called when any Test starts.
  • onTestSuccess- onTestSuccess method is called on the success of any Test.
  • onTestFailure- onTestFailure method is called on the failure of any Test.
  • onTestSkipped- onTestSkipped method is called on skipped of any Test.
  • onTestFailedButWithinSuccessPercentage- method is called each time Test fails but is within success percentage.
  • onFinish- onFinish method is called after all Tests are executed.
How to Implement: there are two ways to implement
  • Class Level
  • Suite Level (in testng.xml)
Lets have a look Class Level first:
1. Class Level

Created Listener class which implements ITestListener interface
package com.TestNGListners;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class ListnerClass implements ITestListener {

 @Override
 public void onTestStart(ITestResult result) {
  
  System.out.println("Test Method " +result.getName() +" Starts");
 }

 @Override
 public void onTestSuccess(ITestResult result) { 
 System.out.println("Test Method " +result.getName() +" Passed"); 
 }
 @Override
 public void onTestFailure(ITestResult result) {
  
  System.out.println("Test Method " +result.getName() +" Failed");  
 }
 @Override
 public void onTestSkipped(ITestResult result) {
  
  System.out.println("Test Method " +result.getName() +" Skipped"); 
 }
 @Override
 public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
  // TODO Auto-generated method stub
 }
 @Override
 public void onStart(ITestContext context) {

  System.out.println("Test " +context.getName() +" Starts");
 }
 @Override
 public void onFinish(ITestContext context) {
 
  System.out.println("Test " +context.getName() +" Ends");
 }
}

Test Class: in this class providing, @Listeners at class level
import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;


@Listeners(ListnerClass.class)

public class TestClass {

 @Test
 public void testCase1() {
  Assert.assertTrue(true);
 }

 @Test
 public void testCase2() {
                Assert.assertTrue(true);

 }
 @Test
 public void testCase3() {
                Assert.assertTrue(false);
 }
 @Test(dependsOnMethods = "testCase3")

 public void testCase4() {
  Assert.assertTrue(true);

 }
}

After executing the above class output will be seen as below:

Output:
Test Default test Starts
Test Method testCase1 Starts
Test Method testCase1 Passed
Test Method testCase3 Starts
Test Method testCase3 Failed
Test Method testCase2 Starts
Test Method testCase2 Passed
Test Method testCase4 Skipped
Test Default test Ends
PASSED: testCase1
PASSED: testCase2
FAILED: testCase3
java.lang.AssertionError: expected [true] but found [false]

Suite Level (in testng.xml)

Test Class: now we are adding one more test class as below
import org.testng.Assert;

import org.testng.annotations.Test;

public class TestClass1 {

 @Test
 public void testCase5() {
  Assert.assertTrue(true);
 }
 @Test
 public void testCase6() {
  Assert.assertTrue(true);
 } 

testng.xml: Configured <listeners> inside suite tab in testng.xml and included both classes (TestClass and TestClass1 in xml below file)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="SmokeSuite">
<listeners>
<listener class-name="com.TestNGListners.ListnerClass"></listener>
</listeners>
  <test name="SmokeTest">
    <classes>
      <class name="com.TestNGListners.TestClass"/>
      <class name="com.TestNGListners.TestClass1"/>
    </classes>
  </test> <!-- SmokeTest -->
</suite> <!-- SmokeSuite -->
}

Output:

Test SmokeTest Starts
Test SmokeTest Starts
Test Method testCase1 Starts
Test Method testCase1 Starts
Test Method testCase1 Passed
Test Method testCase1 Passed
Test Method testCase3 Starts
Test Method testCase3 Starts
Test Method testCase3 Failed
Test Method testCase3 Failed
Test Method testCase2 Starts
Test Method testCase2 Starts
Test Method testCase2 Passed
Test Method testCase2 Passed
Test Method testCase4 Skipped
Test Method testCase4 Skipped
Test Method testCase5 Starts
Test Method testCase5 Starts
Test Method testCase5 Passed
Test Method testCase5 Passed
Test Method testCase6 Starts
Test Method testCase6 Starts
Test Method testCase6 Passed
Test Method testCase6 Passed
Test SmokeTest Ends
Test SmokeTest Ends
===============================================
SmokeSuite
Total tests run: 6, Failures: 1, Skips: 1
===============================================

Please refer below YouTube video on TestNG listeners

Run failed test cases in selenium

At times, test cases may fail while running automated test scripts. The reason may be anything (say, Network issue, System issue or browser issue) but as an automation tester, you need to analyze the result and need to execute the test scripts again. Here is a solution to run failed test cases using TestNG in Selenium.

Why Test fails:
The reason for the failures can be anything
  • Application Failure – not getting expected output
  • Network issue – wifi, LAN issue etc.
  • Server is not responding
  • Scripting issue – Locator change due to new functionality
  • Application is down
  • Browser/browser driver issue 
How to execute failed test cases?
We can execute failed test cases using two methods
  • By running “testng-failed.xml”
  • By Implementing TestNG IRetryAnalyzer interface
Lets have a look the first method:

1. By running “testng-failed.xml"

We have created two classes ClassA and ClassB. Each class has two test methods of which
we are deliberately failing one of the tests. 
public class ClassA {

 @Test

 public void testCase1() {

  System.out.println("this is testcase1");

  Assert.assertTrue(false);

 }
 @Test

 public void testCase2() {
  System.out.println("this is testcase2");

  Assert.assertTrue(true);
 }

}
public class ClassB {

 @Test

 public void testCase3() {

  System.out.println("this is testcase3");

  Assert.assertTrue(true);

 }
 @Test

 public void testCase4() {

  System.out.println("this is testcase4");

  Assert.assertTrue(false);
 }
}

testng.xml
<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Smoke Suite">

  <test name="Test">

    <classes>

      <class name="Com.runFailedTestCases1.ClassA"/>

      <class name="Com.runFailedTestCases1.ClassB"/>

    </classes>

  </test> <!-- Test -->

</suite> <!-- Suite -->

Output
this is testcase1
this is testcase2
this is testcase3
this is testcase4
===============================================
Suite
Total tests run: 4, Failures: 2, Skips: 0

===============================================

If you want to execute only failed test cases through the Eclipse, then first refresh the project. Now navigate to test-output folder and then navigate to Suite folder (in our case suite name is Smoke Suite) Expand that, you will be able to see testng-failed.xml

Open the xml and you will see failed test cases in xml file.
Right-click on this file and click on run as and select the option called "testNG suite".
testng-failed.xml
Suppose if you have three test cases if all the test cases are executed successfully means you are not able to see this folder under the test-output folder. This folder will appear only when one of the test case is failed. Then run this file, it will going to run only failed test cases.

You can create Runner file to execute your failed test cases through script.Please refer below
program.
public class FailTestRunner {

 @AfterTest

 public void runFailTestCases() {

  TestNG obj= new TestNG();

  List<String> list= new ArrayList<String>();

  list.add("D:\\Workspace_Eclipse\\TestNGDemo8\\test-output\\Suite\\testng-failed.xml");

  obj.setTestSuites(list);

  obj.run();
 }
}

2. By Implementing TestNG IRetryAnalyzer interface

Create a class to implement IRetryAnalyzer. Here I am creating a class (say, RetryAnalyzerTest) and implementing IRetryAnalyzer.
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzerTest implements IRetryAnalyzer {

 int count=0;

 int maxCount=2;

 public boolean retry(ITestResult result) {

  if(count<maxCount) {

   System.out.println("Retrying " +result.getName() 

   +" again and count is " +(count+1));

   count++;

   return true;
  }
  return false;
 }

}

Now create another class ‘RetryListnerClass’ by Implementing ‘IAnnotationTransaformer’ interface. transform method is called for every test during test run. A simple implementation of this ‘IAnnotationTransformer’ interface can help us set the ‘setRetryAnalyzer’ for ‘ITestAnnotation’. Add the above class name (RetryAnalyzerTest .class) in the below program. This interface does its work in run time by adding annotation to the test methods.
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;

public class RetryListnerClass implements IAnnotationTransformer {

 public void transform(ITestAnnotation annotation, Class tesClass, 

   Constructor testConstructor, Method testMethod) {

  annotation.setRetryAnalyzer(RetryAnalyzerTest.class);
 }
}

Let us see the example by executing simple tests below
public class ClassA {

 @Test()

 public void testCase1() {
  System.out.println("this is testcase1");
  Assert.assertTrue(true);

 }
 @Test
 public void testCase2() {
  System.out.println("this is testcase2");
  Assert.assertTrue(false);

 }
}

We have two test cases in above program and one of the test case (testCase2) is getting failed.
First lets include below mentioned Listener to testng.xml file. Below mentioned syntax is to add Listener for RetryListnereClass.
<listeners>
<listener class-name="method2.RetryListnerClass"></listener>
</listeners>

Final testng.xml file should looks like below:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Suite">

<listeners>

<listener class-name="method2.RetryListnerClass"></listener>

</listeners>

  <test name="Test">

    <classes>

      <class name="method2.ClassA"/>

    </classes>

  </test> <!-- Test -->

</suite> <!-- Suite -->

Execute the testng.xml. Here is the output which I got. You could see in the below mentioned result that the Test 2 is executed three times as we have mentioned ‘maxCount = 2’. Even though we have just 2 tests, we could find total test runs are 4 in the result.

Output:
this is testcase1
this is testcase2
Retrying testCase2 again and count is 1
this is testcase2
Retrying testCase2 again and count is 2
this is testcase2
===============================================
Suite

Total tests run: 4, Failures: 1, Skips: 2

===============================================

Please refer below YouTube video on how to run failed test cases

Cross Browser Testing in TestNG

What is Cross Browser Testing?
Cross Browser Testing is a type of functional test to check that your web application works as expected in different browsers..

Why do we need?
A web application can be opened in any browser by the end user. For example, some people prefer to open in Firefox, some in chrome, ie etc. We need to ensure that the web application will work as expected in all popular browsers so that more people can access it and use it.

This motive can be fulfilled with Cross Browser Testing of the product.

Some Cross Browser Issues
  • Font size mismatch in different browsers.
  • JavaScript implementation can be different.
  • CSS,HTML validation difference can be there.
  • Some browser still not supporting HTML5.
  • Page alignment and div size.
  • Image orientation.
  • Browser incompatibility with OS.
How to Achieve Cross Browser in Selenium?
To execute test cases with different browsers in the same machine at the same time we can integrate TestNG framework with Selenium WebDriver.

Example: We have created a class CrossBrowserDemo and we have a test method launchApp in it. Using @Parameters annotation we are passing browser name(as string) to test method. And based on the browser parameter our test method launchApp will be executed on particular browser.

Our xml looks like below: Inside <suite> tag we have <parameter name="browser" value="Firefox"></parameter>

Running test on single browser
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="tests">
 <test name="FireFoxTest">
 <parameter name="browser" value="Firefox"></parameter>
  <classes>
   <class name="com.CrossBrowserTest.CrossBrowserDemo"></class>
  </classes>
 </test> <!-- Test -->
</suite> <!-- Suite -->

Running test on multiple browser
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="tests">
    <test name="FireFoxTest">
        <parameter name="browser" value="Firefox"></parameter>
          <classes>
   <class name="com.CrossBrowserTest.CrossBrowserDemo"></class>
         </classes>
</test> <!-- Test -->
 <test name="ChromeTest">
 <parameter name="browser" value="Chrome"></parameter>
  <classes>
   <class name="com.CrossBrowserTest.CrossBrowserDemo"></class>
  </classes>
 </test> <!-- Test -->
 <test name="IETest">
 <parameter name="browser" value="IE"></parameter>
  <classes>
   <class name="com.CrossBrowserTest.CrossBrowserDemo"></class>
  </classes>
 </test> <!-- Test -->
</suite> <!-- Suite -->

 Test Class
public class CrossBrowserDemo {

 WebDriver driver;

 @Parameters("browser")

 @Test

 public void launchApp(String browser) throws InterruptedException {
  if (browser.equalsIgnoreCase("Chrome")) {

   System.setProperty("webdriver.chrome.driver",

     "C:\\Users\\Hitendra\\Downloads\\chromedriver_win32\\chromedriver.exe");

   driver = new ChromeDriver();

  } else if (browser.equalsIgnoreCase("Firefox")) {

   System.setProperty("webdriver.gecko.driver",

     "C:\\Users\\Hitendra\\Downloads\\geckodriver-v0.26.0-win64\\geckodriver.exe");

   driver = new FirefoxDriver();

  } else if (browser.equalsIgnoreCase("IE")) {

   System.setProperty("webdriver.ie.driver",

     "C:\\Users\\Hitendra\\Downloads\\IEDriverServer_x64_3.14.0\\IEDriverServer.exe");

   driver = new InternetExplorerDriver();

  }

  driver.manage().window().maximize();

  driver.get("https://www.automationtestinginsider.com/");

  Thread.sleep(2000);

  driver.close();

 }

}

Output:

Parallel Testing in TestNG

What is a Thread?
Java provides built-in support for multi threaded programming. A multi-threaded program contains two or more parts that can run concurrently. Each part of such a program is called a thread, and each thread defines a separate path of execution.

Parallel Testing
Parallelism or multi-threading in software terms is defined as the ability of the software, operating system, or program to execute multiple parts or sub-components of another program simultaneously.

Benefits of having parallel testing
  • Reduces execution time – As tests are executed in parallel, multiple tests get executed simultaneously, hence reducing the overall time taken to execute the tests.
  • Allows multi-threaded tests – Using this feature, we can write tests to verify certain multi-threaded code in the applications.
There are situations where you want to run multiple tests at the same time.
Situation 1 : If you have 100 test cases and Running these 100 tests sequentially will take you magnitudes of times longer because you can only execute one test at a time.
Situation 2: Cross browser testing - If there are more number of scripts to be executed and executing them on each and every browsers sequentially is time consuming. 
 The above situations can be avoided using a concept in Selenium called Parallel Execution.
 We can reduce the 'execution time' as tests are executed simultaneously in different threads.

In testNG we can achieve parallel execution by two ways.

1. With testng.xml file using parallel attribute of suite tag.
2. We can configure an independent test method to run in multiple threads.

1st Method:
1. With testng.xml file using parallel attribute of suite tag.
Syntax: parellel =“methods”
The parallel attribute of suite tag can accept four values: methods ,classes, tests, instances
Sample XML <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel=“methods">
  <test name="Test">
    <classes>
      <class name="com.TestPackage.ClassB"/>
    </classes>
  </test> <!-- Test -->
</suite> <!-- Suite -->

Example: I am giving an example here of parallel testing by methods, rest you can watch below youtube video. In below example 4 methods from two classes will be executed in two threads (thread-count="2").
public class ClassA {

 @Test

 public void testcase1() {

  System.out.println("This is testcase1: "+Thread.currentThread().getId());

 }
 @Test

 public void testcase2() {

  System.out.println("This is testcase2: "+Thread.currentThread().getId());
 }
}

public class ClassB {

 @Test
 public void testCase3() {

  System.out.println("This is testcase3: "+Thread.currentThread().getId());

 }
 @Test

 public void testCase4() {

  System.out.println("This is testcase4: "+Thread.currentThread().getId());
 }
}

TestNG.xml
<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">

<suite name="Suite" parallel="methods" thread-count="2">

  <test name="Test">

    <classes>

      <class name="com.parallelDemo.ClassA"/>

      <class name="com.parallelDemo.ClassB"/>

    </classes>

  </test> <!-- Test -->

</suite> <!-- Suite -->

Output:
This is testcase2: 14
This is testcase1: 13
This is testcase3: 14
This is testcase4: 13

2nd Method: 2. We can configure an independent test method to run in multiple threads.

We can configure test method like below and in classC test method testMethod1 will be executed in
two threads (threadPoolSize = 2) and will be executed 3 times(invocationCount = 3)  
public class ClassC {

 @Test(threadPoolSize = 2, invocationCount = 3, timeOut = 1000)

 public void testMethod1() {

  System.out.println("This is testcase1: "+Thread.currentThread().getId());

 }
}

Output:
This is testcase1: 11
This is testcase1: 12
This is testcase1: 11

Please refer below YouTube video on Parallel Testing: