Explaining the Well-Known Path Prefix

Before making a request to a website, web protocols are increasingly requiring the discovery of what's called "site-wide" metadata. This metadata essentially provides policy or other information pertaining to that particular site. The well-known path prefix is used for that type of information to be stored so that it is easily retrievable.

In this post, we'll go into more detail regarding what the well-known path prefix actually is, how to use it, etc.

What is the well-known path prefix?

The well-known path prefix is essentially a place where you should store information about a site that is expected to be at the same location on every website. The well-known prefix is defined as follows: /.well-known/. What should be included after the well-known prefix is defined in the following registry (more details on this below).

Therefore, if your website has a URI for acme-challenge, then according to the registry, this URI should be defined within the well-known path prefix as follows:

/.well-known/acme-challenge

The well-known concept was created to provide a central location for these metadata resources to live. Otherwise, if users were to just create their own well-known paths (a common one being /robots.txt) then this would become cumbersome to keep track of and resources may collide with one another as more are created.

It should be noted that robots.txt does not fall within the list of resources to include within the /.well-known/ directory. Robots.txt was created long before well-known and therefore it does not apply.

Using well-known URIs

Using well-known URIs is fairly straightforward. Basically, if you have resources that are included within the well-known registry list then you should move them to live within the well-known path prefix. As RFC 5785 states:

The well-known URI space was created with the expectation that it will be used to make site-wide policy information and other metadata available directly (if sufficiently concise), or provide references to other URIs that provide such metadata.

It could very well be that you do not require the use of well-known URIs depending upon your site's setup. Additionally, even if you have never setup or used any well-known URIs you may still see them appear in Google Webmaster tools. This is due to the fact that the Googlebot is crawling your site to verify whether or not the files exist. If they don't, the crawler will return a 404 for those files.

This is nothing to be worried about. It is simply stating that the crawler checked to see whether the files existed and when it couldn't find them, it returned a 404 Not Found.

Well-known registry

The well-known URIs registry contains a list of all URI suffixes that should be included within the path prefix: ./well-known/. If you're using files that you believe should be found at the same place for all websites, then check the registry.

If you have a URI that you think should be on the well-known registry list, you'll need to send a request to IETF's mailing list for review. Your request will then be evaluated by a team of experts and you should hear back from them before a period of 14 days to determine whether or not your request has been approved or denied.

Summary

Although the well-known path prefix isn't something that all webmasters need to implement, it is useful to know what it is and how to use it. In the event that you are providing certain site-wide metadata information, it's important to check the complete well-known registry list to verify whether or not your URIs should be located under the /.well-known/ path prefix.