Google to open-source Robots.txt Projects for C++ and Java

Share on facebook
Share on google
Share on twitter
Share on linkedin
A VPN is an essential component of IT security, whether you’re just starting a business or are already up and running. Most business interactions and transactions happen online and VPN


  • Open Sourcing robot.txt projects for C++ and Java  
  • Specification Test for robots.txt  
  • Java port of C++ robots.txt parser  
  • Requirements for new testing frameworks  

Previously, Google had open-sourced robots.txt that is used in Java. However, now it has announced to release some additional source code projects in C++ and Java.   

C++ and Java:  

Google has officially released the source code that validates the robots.txt testing framework using C++. It ensures that the outcomes after parsing (parser results) are following the official robots.txt specifications. Moreover, it also validates parsers written in different languages other than C++. Besides C++, Modern Java is a widely used language in various applications. Google has announced to release the official port Java as well.   

In its blog post, Google writes:  

“Last year we released the robots.txt parser and matcher that we use in our production systems to the open source world. Since then, we’ve seen people build new tools with it, contribute to the open source library (effectively improving our production systems- thanks!), and release new language versions like golang and rust, which make it easier for developers to build new tools.”  

Robots.txt Specification Test:  

Google announced on Monday that they are officially releasing a testing framework, created by Andreea for the robots.txt parser developers.  

The blog post states:   

“Currently there is no official and thorough way to assess the correctness of a parser, so Andreea built a tool that can be used to create robots.txt parsers that are following the protocol.”  

Java robots.txt parser and matcher:  

In addition to this, Ian created the Java port of C++ robots.txt parser. Speaking of functions and behaviours, the new parser is an exact duplicate of C++ parser. Teams are very eager to use the Java robots.txt parser in production systems, and let’s hope that they find it convenient.   

The requirement to Run the Testing Frameworks:  

  • Java Development Kit 1.7+ for Apache and Maven (Automation tool for Java projects)  
  • Google’s protocol buffers  

As always, Google invites the developers to contribute to these projects and let them know if there is a room for improvement and development.   


Nisha Zahid

Nisha Zahid

Nisha is a research writer having keen interest in technology. According to her digital advancement is the necessity of the hour. Therefore, she keeps herself abreast of all the latest developments happening in this field. In the free time, she loves to read books.

Leave a Replay

Sign up for our Newsletter

Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit