Priyanshu Nayan

Google Summer of code 2020 Report

As a participant of the Google Summer of Code Program, I worked with an open source organisation called Hydra Ecosystem.

Hydra is a documentation framework that bases on top of established Linked Data (Semantic Web) tools to build the next generation of connected Web APIs. The team of developers in the Hydra Ecosystem aims to establish an ecosystem of tools to make Hydra framework operational.

General Improvements in Agent

In the course of the past three months I worked on general improvements in the agent and the core library and brought them on par with the new developments in hydra specification. I worked on hydra-python-core that is responsible for the implementation of the hydra specification which is then used by the server(hydrus) and the smart agent. I also worked on the agent to bring it on par with the changes in the core library as well as server. I also worked on the GUI of the smart agent that automatically discovers all the possible endpoints and the properties of any hydra aware API.

The module dependency diagram is given below:

Here are all the implementations and improvements (in the form of PR) from the last three months.

PHASE 1

Automating the discovery of API Documentation:

Instead of hardcoding ‘vocab’ after entry point url, this PR retrieves api doc url from the Link header according to the Hydra spec. The API documentation must be present in the Link Header with rel hydra:apidocumentation. The corresponding PR: Discover api-doc url from Link header by priyanshunayan · Pull Request #145 ·… *Fixes #102 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Changing the UI/UX of the agent GUI, and hosting them on GCP

We used material design principles to create UI components. The GUI now is responsive, displays important information regarding the Classes and Collection and the properties like if they are optional or required. This is important for beginners to get an idea, what sort of APIs they are dealing with. Also one can now play with their APIs right from their smartphone. Corresponding PR: Improve UI by priyanshunayan · Pull Request #62 · HTTP-APIs/hydra-python-agent-gui *input component styled change endpoints and operation button styles style properties editor raw command and input props…*github.com

Implementation of Paginator in agent:

Expose multiple interfaces such as moving forward one at a time or going back once at a time in the agent. Also, in many cases, developers need to know things like the number of pages and sometimes they might want to jump to last page or to a specific page programmatically. Corresponding PR: Add partial collection Paginator by priyanshunayan · Pull Request #147 ·… *Fixes # Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

IRI Template Expansion Strategy

The client might need to construct the URL based on parameters retrieved on runtime. This construction can take place in two ways, Basic: where the template returned by the server follows the IRI rules defined in the w3c spec and Explicit: where other meta information can be passed along like language of the parameters, type of params(decimal, string etc) etc. Thus we implemented both Basic and Explicit Expansion strategies of templates in the agent. Corresponding PR: Templated IRI Support by priyanshunayan · Pull Request #146 · HTTP-APIs/hydra-python-agent *Fixes # Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Implementation of Pagination and Templated Links in GUI

Since GUI is based on the agent library, and agent now supported Pagination and templated Links so we demonstrated both the features on the GUI. It’s corresponding PR: Improve UI by priyanshunayan · Pull Request #62 · HTTP-APIs/hydra-python-agent-gui *Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes…*github.com

Blogs

Phase 2

Identification of collections and classes in entrypoint

The entrypoint is the place where the agent discovers all the endpoints. So, it makes sense to differentiate class endpoints and collection endpoints in the entrypoint so that agent can take wise decisions automatically on the basis of it. So we added the hydra:collection array in the entry-point. Also we added the default manages block which gives the implicit information about the members of the collection. It’s corresponding PR: Add collection list and manages block by priyanshunayan · Pull Request #40 ·… *Addresses #39 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python…*github.com

Expansion of JSONLD and parsing of API DOC

Earlier we used to parse apidoc in the raw form. But that did not give flexibility of parsing different types of JSON-LD document. So, we decided to use the Expansion algorithm that expands the document and makes it contextless and programmatically it’s easier to parse absolute IRIs. As a result the code to parse the docs became a lot simpler. Corresponding PR with relevant tests: Expansion of JSONLD and parsing of API DOC by priyanshunayan · Pull Request #42 ·… *Related #38, #37 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python…*github.com

Removed inter dependency of classes and collection

A collection is defined as a set of somehow related resources, not necessarily a set of resources of particular class. So, we removed the interdependency of classes and collection. Corresponding PR: Remove dependency of class collection by priyanshunayan · Pull Request #45 ·… *Fixes #44 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Minor Bug fix in hydrus server

Returned the correct template syntax as mentioned in the spec and fixed the issue of throwing error by the sever if no pagination flag was enabled. correct template and fix no-pagination by priyanshunayan · Pull Request #486 · HTTP-APIs/hydrus *Fixes #485 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Blogs

Phase 3

Bug fixes in core library

This PR resolves all the bugs that crept during the development in core. Bug Fixes by priyanshunayan · Pull Request #48 · HTTP-APIs/hydra-python-core *Related: HTTP-APIs/hydrus#482 Checklist My branch is up-to-date with upstream/develop branch. Everything works and…*github.com

Updated Sample Docs

The sample docs that we use in the server and agent needed to be updated in hydrus. Update sample docs by priyanshunayan · Pull Request #491 · HTTP-APIs/hydrus *Fixes #482 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Configure sphinx documentation for core

The core being one of the most important library in the ecosystem it lacked proper documentation, so we hosted the documentation on readthedocs.io and ensured continuous documentation Add Docs by priyanshunayan · Pull Request #47 · HTTP-APIs/hydra-python-core *Fixes # Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Generalize Sync Mechanism

Earlier the agent used to work only with hydrus and supported the sync mechanism, but not all servers support sync mechanism, hence agent should identify which servers it should connect with sync and communicate with normal HTTP request if the server is not hydrus. Generalise sync mechanism by priyanshunayan · Pull Request #154 · HTTP-APIs/hydra-python-agent *Fixes #141 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Bringing on par with latest changes in hydrus and core library

The first two phases saw changes in hydrus as well as core library, hence to bring on par with hydrus and core. This PR enables collection as a resource in agent. All the instances of classes are added to their corresponding class node not the collection node. The collection instances are not cached, and the agent returns the HTTP response as it is. Corresponding PR: Support Collection as a resource in agent and bring on par with latest changes in core library by… *Fixes #152 Checklist My branch is up-to-date with upstream/develop branch. Everything works and tested for Python 3.5.2…*github.com

Blogs

Documentation Tasks:

Auxiliary PRs:

Learnings:

  • Hydra ecosystem and REST APIs in General

Hydra is a tool to build and document APIs. So, to implement the specification into our project I went through their bug trackers, read their documents and tried to understand why the decision was taken. This helped me learn a lot about RESTful APIs, all the necessary building blocks of any API like pagination, templated links, collections, entrypoint etc. I realized the most overlooked property of RESTful API is the hypermedia. So, we made sure we use proper IRIs or its templates to link resources together forming the basis of linked data aka semantic web.

  • Working on a larger than usual codebase.

My project involved work across different repositories, so initially I had a little bit of difficulty traversing around the project. I realized how important it was to understand the build process of a project, the design pattern the project follows, the entrypoint of the project, its dependencies etc. Once these things are figured out, implementing methods or classes is not difficult afterwards.

  • Advanced Git Operations

Working together as a team requires solid understanding of tools.. The Github way of contributing by opening Pull requests is really helpful to collaborate together. In the process of collaborating and using Git, I also realised the importance of commit messages.. The PRs should be comprehensible with proper descriptions. Since multiple people were working on the projects we needed to rebase our PRs to the latest developments of the projects. Doing this over and over again helped me form a habit of reviewing, adding and committing changes.

  • Working as a team

Open source projects require collaboration. We worked together as a team to squash a number of bugs together. We split our tasks into smaller ones and divided it among ourselves. This increased the work process and helped us to finish the large tasks easier.

  • Programming in Python and Javascript

My project involved a lot of Python and Javascript.. We followed Object Oriented Methodology in our projects. This helped us get rid of many repeated codes and thus helped us follow the DRY Principle.. We also wrote unit tests to make sure everything worked as expected.Unit tests would also prevent unwanted changes in the PRs — thanks to continuous integration.

  • Writing Documentation

There is a popular saying that goes like this: “Docs, or it didn’t happen” . So we followed documentation guidelines laid down by our organization to document our code. It’s a no secret that document plays an important role in success of any project, hence we wrote comprehensive documentation and also ensured continuous documentation using readthedocs.org. This will ensure wider adoption as people would be able to understand the tools even better.

Acknowledgements:

This project was done for the Hydra Ecosystem Organization in Google Summer of Code program under the mentorship of

  1. Chris Andrew

  2. Akshay Dahiya

  3. Lorenzo

They provided much needed support and guidance wherever we were stuck.

I would also like to mention Samesh, and thank him for his eagerness to collaborate and discuss tasks during the program.

I post more frequently on Twitter. Follow me there to see what I am upto these days!