Use Github As a Content Source

SearchUnify can crawl, index, and search your GitHub repositories. This article gets you started.

PERMISSIONS.

SearchUnify ignores user permissions during searches. All indexed files can be search by all users.

PREREQUISITES

You should have access to the repositories to be crawled.

Establish a Connection

1. Navigate to Content Sources and click Add New Content Sources.

   

2. Find GitHub and click Add.

   

3. Under the Authentication tab, enter the required details:

• Give your content source a Name.

• Pick either No Authentication or OAuth from Authentication Type dropdown.

  Note. Only the issues from public repositories are crawled by default. If you want to crawl private repositories, please contact SearchUnify support.

• Select the comments language in Language.

• Select the type of your GitHub account in Repository Type. Individual developers can select User but the correct way forward for companies is Organization.

• Label your repository in Organization/User Name.

  

4. After entering the aforementioned details, click Connect.

Click Next if a window pops up with the "Connection Successful" message. Once the connection has been set up successfully, you will be prompted to the next action - Set Frequency.

Re-Connect

The Authentication screen is displayed when an existing Content Source is opened for editing, as given below. An admin can edit a Content Source for multiple reasons:

• To reauthenticate

• To fix a crawl error

• To change frequency

• To add or remove an object or a field for crawling

• For multiple other reasons

When you edit a Content Source, there can be any one of two cases as listed below:

If the Content Source authentication is successful; a Connected message is displayed.

Case 1: There are no crawl errors and the Content Source authentication is valid.

If the Content Source authentication fails or is disrupted; a Re-Connect button is displayed.

There is a crawl error or the authentication details have changed. In both cases, the SearchUnify Content Source connection must be authenticated again i.e. re-authenticated. To authenticate a Content Source again, enter the authentication details, and click Re-Connect.

Set Up Crawl Frequency

The first crawl is always manual and is performed after configuring the content source. In Choose A Date, select a date to start crawling; the data created after the selected date will be crawled. For now, keep the frequency to its default value Never and click Set and move to the next section.

Select Fields and Repositories for Indexing

SearchUnify indexes one GitHub content type, issues. An admin can decide which issues fields are to be indexed. Under the Rules tab, you will land on By Content Type subtab.

1. Click to view the pre-configured properties of issues.

   

NOTE. You can add or delete the content fields. Although, it is not recommended for users other than Admins to make any changes in the fields.

2. Navigate to By Spaces, use the index to find and select the repositories you want to crawl. If you don't specify repositories, then all issues of your GitHub repositories are indexed.

   Note. Both open and closed issues are crawled in SearchUnify.

   

3. Save your Settings.

You have successfully added GitHub as a content source in SearchUnify. Perform a manual crawl to start indexing data in SearchUnify.

Related

Difference between Manual and Frequency Crawls

After the First Crawl

Return to the Content Sources screen and click in Actions. The number of indexed documents is updated after the crawl is complete. You can view crawl progress in in Actions. Documentation on crawl progress is in View Crawl Logs.

Once the first crawl is complete, click in Actions open the content source for editing, and set a crawl frequency.

1. In Choose a Date, click to fire up a calendar and select a date. Only the data after the selected date is indexed.

   

2. The following options are available for the Frequency field:

   • When Never is selected, the content source is not crawled until an admin opts for a manual crawl on the Content Sources screen.

   • When Minutes is selected, a new dropdown appears where the admin can choose between four values: 2, 15, 20, and 30. Picking 20 means that the content source crawling starts every 20 minutes.

   • When Hours is selected, a new dropdown is displayed where the admin can choose between eight values between 1, 2, 3, 4, 6, 8, 12, and 24. Picking 8 means that the content source crawling starts every 8 hours.

   • When Daily is selected, a new dropdown is displayed where the admin can pick a value between 0 and 23. If 15 is chosen, then the content source crawling starts at 03 p.m. or 1500 hours every single day.

   • When Day of Week is selected, a new dropdown is displayed where the admin can pick a day of the week. If Tuesday is chosen, then content source crawling starts at 0000 hours on every Tuesday.

   • When Day of Month is selected, a new dropdown appears where the admin can select a value between 1 and 30. If 20 is chosen, then content source crawling starts on the 20th of each month.

     It’s recommended to pick a date in the range 1-28. If 30 is chosen, then the crawler may throw an error in February. The error will be “Chosen date will not work for this month.”

   • When Yearly is selected, the content source crawling starts at midnight on 1 January each year.

   

3. Click Set to save crawl frequency settings. On clicking Set, you are taken to the Rules tab.

   

Data Deletion and SU Index

A method to update the index in real time is to enable event subscriptions, which supplement existing crawls and synchronize data between your GitHub instance and SearchUnify in real time. Check out Enable Event Subscription in GitHub.