Websites being the primary source of communication in the digital transformation world, have evolved humongously since the last decade. Web development has grown at a tremendous pace with lots of automation testing frameworks coming in for both frontend and backend development. Websites have become smarter and so have the underlying tools and frameworks. With a significant surge in the web development area, the browsers have also become smarter. Nowadays, you can find headless browsers, where users can interact with the browser without GUI. You can even scrape websites in headless browsers using packages like Puppeteer and nodejs. 

Efficient web development hugely relies on a testing mechanism for quality assessment before we can push them in production environments. Headless browsers can perform end to end testing, smoke testing etc at a faster speed as it is free from overhead of the memory space required for the UI. Moreover, studies have proved that the headless browsers generate more traffic than the non-automated ones. Popular browsers like Chrome can even help in debugging the web pages in real time, analyse performance, notify the memory consumption, enable tweaking of the code and analyse performance in real time etc. Isn’t this evolution of browsers heading towards a smarter web development process? So in this blog we will have an overview on headless browsers and understand how it helps in smarter and faster website development.

 

What is a Headless Browser ?

 

 

A headless browser is simply a browser without the GUI. It has got all the capabilities of rendering a website, like a normal website. Since GUI is not available in the browser with headless mode, we need to use the command line to interact with the browser. Headless browsers are designed for tasks like automation testing,  javascript library testing, javascript simulation and interactions. 

One of the biggest reasons for using headless browser or headless browser testing is that it let’s you run the tests more quickly and in a real environment. For eg, the combination of chrome devtools and headless chrome lets you edit pages on the fly and, which helps you in diagnosing the problem quickly ultimately helping you in developing better websites faster. So headless browsers are more fast, flexible and optimised in performing tasks like web-based automation testing. Like a normal browser, the headless browser is capable of performing tasks like links parsing JavaScript, clicking on links, coping with any downloads and for executing this we need to use command line. So it can provide a real browser context without any of the memory consumed for running a full-fledged browser without a GUI.

Comparative Analysis Between Headless Browsers and Traditional Browsers

Traditional Browsers
Traditional browsers like Google Chrome, Mozilla Firefox, Internet Explorer, and Safari are designed with the end-user in mind. They come with a graphical user interface (GUI) to allow users to interact with web applications directly.
Speed: Traditional browsers are slower because they must render all webpage elements, including images, CSS, and scripts, which can be resource-intensive.
Automation & Testing: Automating tests can be more challenging with traditional browsers because of their heavy reliance on the UI. They often require additional tools like Selenium to perform automation testing.
Resource Usage: They consume more resources (CPU, memory) as they have to render GUI along with webpage content.
Web Scraping: Web scraping in traditional browsers is not as efficient because of their GUI and rendering of all webpage content.

Headless Browsers
Headless browsers, on the other hand, are designed with developers and automation in mind. They do not have a GUI, and interactions are through a command line or with the help of code.
Speed: As headless browsers do not render GUI, they load pages faster. This speed is critical for automated testing where multiple test cases are executed in parallel.
Automation & Testing: Headless browsers are perfect for automation as they provide direct interaction with the HTML of a webpage. Tools like Puppeteer make automation testing even simpler.
Resource Usage: Since they don’t have a GUI to render, headless browsers consume less memory and CPU, making them more efficient.
Web Scraping: Web scraping is efficient and faster in headless browsers as they can render the webpage and access the HTML directly.

 

The Need for a Headless Browser 

 

 

With advancements in the website development technologies, website testing has taken a center stage and emerged as the most essential steps in developing high performing websites. Even browsers are becoming smarter as they can load the javascript libraries for performing automation testing. Isn’t that a significant transformational  leap in the website testing. So let’s have an overview on some of the major functions performed by headless browsers.

Enables faster web testing using Command Line Interface

 

 

With headless cross browser testing, we are saved from the overhead of memory consumed in GUI, hence it enables faster website testing, using command line as the primary source of interaction. The headless browsers are designed to execute crucial test cases like end to end testing which ensures that the flow of an application is performing as designed from start to finish. The headless browsers cater to this use case as they enable faster website testing. 

Scraping websites

 

The headless browser saves the overhead of opening the GUI thus enabling faster scraping of the websites. In headless browsers we can automate the scraping mechanism and extract the data in a much more optimised manner. 

Taking web screenshots 

 

Though the headless browsers do not avail any GUI, they do allow the users to take snapshots of the website that they are rendering. It’s very useful in cases where the tester is testing the website and needs to visualise the code effects and save results in the form of screenshots. In a headless browser you can easily take a large number of screenshots without any actual UI.

Mapping user journey across the websites 

 

Headless browsers allow you to programmatically map the customer journey test cases. Here headless browsers help the users to optimise the user experience throughout their decision making journey on the website.

Now that we have understood what is a headless browser and it’s numerous features, along with the its key quality of being a lightweight browser which helps in accelerating the speed of testing, let’s have an overview on the most popular headless browser, Headless Chrome and see what does it unlocks  

 

Diving into Headless Chrome and Chrome DevTools

 

We have a number of headless browsers and to name a few are firefox version 55 and 56, PhantomJs, Html Unit, Splinter, jBrowserDriver etc. Chrome 59 is Chrome’s version to run it in headless mode. Headless Chrome and Chrome DevTools is quite a powerful combination enabling the users with out of box features. So let’s have an overview on Headless Chrome and Chrome DevTools.

 

What is Headless Chrome

 

Headless Chrome shipping in Chrome 59, is basically running chrome in a headless environment. It’s running chrome without GUI. This light weighted, memory sparing and quick running browser brings in all the modern web platform features provided by the chromium and blink rendering engine to the command line.

As per the studies, the automated browsers always generated more traffic than the non-automated ones. In a recent survey, it was discovered that the headless Chrome generated more traffic than its previous leader Phantum Js within a year of its release. 
Apart from this, there are several reasons why chrome is the most popular headless browser. One of the reasons is, it’s always updating out of the box features, which constantly introduce new trends in web development. It also consists of a rendering engine called Blink , which constantly updates itself as the website evolves. What does the Headless Chrome Unlocks :

• The ability to test the latest web platform features like ES6 modules, service worker and streams 
• Enables to programmatically tap into the chrome devtools and makes use of the awesome features like network throttling, device emulating, website performance analysis etc. 
• Test multiple levels of navigation
• Gather page information
• Take screenshots
• Create PDFs

Now let’s have a look on the most common flags to start working with headless chrome:

Starting headless

 

For starting headless, you need a Chrome 59+ and open the chrome binary from the command line. If you have got the Chrome 59+ installed then start the Chrome with

 

The — headless flag starts the chrome in headless mode 

 

Similarly for printing the DOM, creating pdf, taking screenshots we can simply use the following flags 

Printing the DOM

The –dump-dom flag prints document.body.innerHTML to stdout

Create a PDF

The –print-to-pdf flag creates a PDF of the page:

Taking screenshots

To capture a screenshot of a page, use the –screenshot flag

 

Debugging a code without the browser UI

If you want to debug your code in a headless browser using the chrome devtools then make note of the following flag.–remote-debugging-port=9222. This flag helps you to open the headless chrome in a special mode, wherein the chrome devtool can interact with the headless browser to edit the web page during run-time. We will dig deeper into Chrome Devtools in the later section of the blog.

For debugging the web page with chrome devtools, use the –remote-debugging-port=9222 flag.

 

What is Chrome DevTool ?

Chrome DevTools is a set of web developer tools built directly into Google Chrome. It helps to debug the web pages on the fly and hence helps to detect the bugs quickly, which ultimately helps to develop the websites faster. 

The simplest way of opening devtools is, right click on your webpage and click inspect. Now based on your purpose to use the devtool, you can open various consoles. For example to work with DOM or CSS you can click on the elements panel, to see logged messages or run javascript click on the console panel, for debugging the javascript click on the source panel, to view network activity click on the network panel, to analyse the performance of the webpage click on the performance panel, to fix memory problems click on the memory panel.

As we can see, the chrome devtools is a package of diverse functionalities which helps in debugging a web page in the chrome browser. But what about the headless browser with no UI, how can we debug the web page with no UI?. Can chrome devtools help debugging a headless browser ? Let’s demystify the ways of debugging a headless browser with chrome devtools, discuss what is puppeteer in the following sections of the blog.

 

Puppeteer

As discussed earlier, one of the ways of debugging a web page in a headless browser is to use the flag  –remote-debugging-port=9222 in the command line which helps in tapping into the chrome devtools programmatically. But, there is another layer into the picture to play with the headless chrome to perform numerous out of the box tasks and make use of headless in a more efficient way. Here, Puppeteer comes into the picture.

Puppeteer is a Node Library which provides a high level API to control chrome over devtools protocol. Puppeteer is usually headless but can also be configured to use full non – headless Chrome.  It provides full access to all the features of headless chrome and can also run chrome fully in a remote server which is very beneficial to the automation teams. It would be quite justifying to term Puppeteer as Google Chrome team’s official Chrome headless browser.

One of the greatest advantages of using puppeteer as an automation framework for testing is that unlike other frameworks, it is very simple and easy to install. 

As puppeteer is a node javascript library, so first of all you need to install nodejs on your system. Nodejs comes with the npm (node package manager) which will help us to install the puppeteer package.

The following code snippet will help you to install nodejs

## Updating the system libraries

##sudo apt-get update

## Installing node js in the system 

##sudo apt-get install nodejs 

Once you are done with installation of node js in your machine, you can run the following flag to install puppeteer.

npm i puppeteer

With this you completed the installation for puppeteer which will also by default download the latest version of chrome. 

 

Why is puppeteer so useful ?

Puppeteer provides full access to all the out of box features provided by the headless chrome and its constantly updating rendering engine called blink. Other than mostly used automation testing frameworks for web applications like selenium web driver, puppeteer is so popular as it provides automation for a light weighted (UI less) headless browser which helps in testing faster. Likewise there are multiple functionalities provided by puppeteer, so let’s have a look at it 

• It can help generate screenshots and pdfs of pages 
• Crawl a single page application and generate pre-rendered content 
• Automate form submission, UI testing, end to end testing, smoke testing, keyboard input etc 
• Creates an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
• Captures a timeline trace of your site and analyses performance issues.
• It can test Chrome Extensions.
• Allows performing web scraping 

As we are aware of the functionalities performed by puppeteer let’s have an overview on the code snippet for taking screenshot in puppeteer.  

const puppeteer = require(‘puppeteer’);

(async () => {

const browser = await puppeteer.launch();

const page = await browser.newPage();

await page.goto(‘https://example.com’);

await page.screenshot({path: ‘example.png’});

await browser.close();

})();

Once this code gets executed, a screenshot will be saved in your system through the path mentioned in the code snippet.

Conclusion

Faster and qualitative web development has always been and will always be the top priority of QA and development teams. The headless browsers ( without GUI) being light weighted and memory sparing runs at a higher speed while automation testing. Definitely they cater to the need of smarter web development. Moreover, they help in testing all the modern web platform features as well as enable debugging and performance analysis during real time which adds another feather on the cap. They are responsible for heavy traffic in the web applications and support website scraping with the help of packages like nodejs and puppeteer. Furthermore, the installation of headless browsers is easier than installation of any other web automation frameworks like selenium. These qualities of headless browsers are compelling the brands to utilize them for developing high performance web applications.