You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

244 lines
8.9 KiB

6 years ago
5 years ago
5 years ago
6 years ago
  1. XRay
  2. ====
  3. XRay parses structured content from a URL.
  4. ## Discovering Content
  5. The contents of the URL is checked in the following order:
  6. * A silo URL from one of the following websites:
  7. ** Instagram
  8. ** Twitter
  9. ** GitHub
  10. ** XKCD
  11. ** (more coming soon)
  12. * h-entry, h-event, h-card
  13. ## Parse API
  14. To parse a page and return structured data for the contents of the page, simply pass a url to the parse route.
  15. ```
  16. GET /parse?url=https://aaronparecki.com/2016/01/16/11/
  17. ```
  18. To conditionally parse the page after first checking if it contains a link to a target URL, also include the target URL as a parameter. This is useful when using XRay to verify an incoming webmention.
  19. ```
  20. GET /parse?url=https://aaronparecki.com/2016/01/16/11/&target=http://example.com
  21. ```
  22. In both cases, the response will be a JSON object containing a key of "type". If there was an error, "type" will be set to the string "error", otherwise it will refer to the kind of content that was found at the URL, most often "entry".
  23. You can also make a POST request with the same parameter names.
  24. ### Authentication
  25. If the URL you are fetching requires authentication, include the access token in the parameter "token", and it will be included in an "Authorization" header when fetching the URL. (It is recommended to use a POST request in this case, to avoid the access token potentially being logged as part of the query string.) This is useful for [Private Webmention](https://indieweb.org/Private-Webmention) verification.
  26. ```
  27. POST /parse
  28. url=https://aaronparecki.com/2016/01/16/11/
  29. &target=http://example.com
  30. &token=12341234123412341234
  31. ```
  32. ### Twitter Authentication
  33. XRay uses the Twitter API to fetch posts, and the Twitter API requires authentication. In order to keep XRay stateless, it is required that you pass in Twitter credentials to the parse call. You can register an application on the Twitter developer website, and generate an access token for your account without writing any code, and then use those credentials when making an API request to XRay.
  34. You should only send Twitter credentials when the URL you are trying to parse is a Twitter URL, so you'll want to check for whether the hostname is `twitter.com` before you include credentials in this call.
  35. * twitter_api_key - Your application's API key
  36. * twitter_api_secret - Your application's API secret
  37. * twitter_access_token - Your Twitter access token
  38. * twitter_access_token_secret - Your Twitter secret access token
  39. ### Error Response
  40. ```json
  41. {
  42. "error": "not_found",
  43. "error_description": "The URL provided was not found"
  44. }
  45. ```
  46. Possible errors are listed below:
  47. * `not_found`: The URL provided was not found. (Returned 404 when fetching)
  48. * `ssl_cert_error`: There was an error validating the SSL certificate. This may happen if the SSL certificate has expired.
  49. * `ssl_unsupported_cipher`: The web server does not support any of the SSL ciphers known by the service.
  50. * `timeout`: The service timed out trying to connect to the URL.
  51. * `invalid_content`: The content at the URL was not valid. For example, providing a URL to an image will return this error.
  52. * `no_link_found`: The target link was not found on the page. When a target parameter is provided, this is the error that will be returned if the target could not be found on the page.
  53. * `no_content`: No usable content could be found at the given URL.
  54. * `unauthorized`: The URL returned HTTP 401 Unauthorized.
  55. * `forbidden`: The URL returned HTTP 403 Forbidden.
  56. ### Response Format
  57. ```json
  58. {
  59. "data":{
  60. "type":"entry",
  61. "published":"2017-03-01T19:00:33-08:00",
  62. "url":"https://aaronparecki.com/2017/03/01/14/hwc",
  63. "category":[
  64. "indieweb",
  65. "hwc"
  66. ],
  67. "photo":[
  68. "https://aaronparecki.com/2017/03/01/14/photo.jpg"
  69. ],
  70. "syndication":[
  71. "https://twitter.com/aaronpk/status/837135519427395584"
  72. ],
  73. "content":{
  74. "text":"Hello from Homebrew Website Club PDX! Thanks to @DreamHost for hosting us! 🍕🎉 #indieweb",
  75. "html":"Hello from Homebrew Website Club PDX! Thanks to <a href=\"https://twitter.com/DreamHost\">@DreamHost</a> for hosting us! <a href=\"https://aaronparecki.com/emoji/%F0%9F%8D%95\">🍕</a><a href=\"https://aaronparecki.com/emoji/%F0%9F%8E%89\">🎉</a> <a href=\"https://aaronparecki.com/tag/indieweb\">#indieweb</a>"
  76. },
  77. "author":{
  78. "type":"card",
  79. "name":"Aaron Parecki",
  80. "url":"https://aaronparecki.com/",
  81. "photo":"https://aaronparecki.com/images/profile.jpg"
  82. }
  83. },
  84. "url":"https://aaronparecki.com/2017/03/01/14/hwc",
  85. "code":200
  86. }
  87. ```
  88. #### Primary Data
  89. The primary object on the page is returned in the `data` property. This will indicate the type of object (e.g. `entry`), and will contain the vocabulary's properties that it was able to parse from the page.
  90. If a property supports multiple values, it will always be returned as an array. The following properties support multiple values:
  91. * in-reply-to
  92. * like-of
  93. * repost-of
  94. * bookmark-of
  95. * syndication
  96. * photo (of entry, not of a card)
  97. * video
  98. * audio
  99. * category
  100. The content will be an object that always contains a "text" property and may contain an "html" property if the source documented published HTML content. The "text" property must always be HTML escaped before displaying it as HTML, as it may include unescaped characters such as `<` and `>`.
  101. The author will always be set in the entry if available. The service follows the [authorship discovery](http://indiewebcamp.com/authorship) algorithm to try to find the author information elsewhere on the page if it is not inside the entry in the source document.
  102. All URLs provided in the output are absolute URLs. If the source document contains a relative URL, it will be resolved first.
  103. In a future version, replies, likes, reposts, etc. of this post will be included if they are listed on the page.
  104. ```json
  105. {
  106. "data": {
  107. "type": "entry",
  108. ...
  109. "like": [
  110. {
  111. "type": "cite",
  112. "author": {
  113. "type": "card",
  114. "name": "Thomas Dunlap",
  115. "photo": "https://s3-us-west-2.amazonaws.com/aaronparecki.com/twitter.com/9055c458a67762637c0071006b16c78f25cb610b224dbc98f48961d772faff4d.jpeg",
  116. "url": "https://twitter.com/spladow"
  117. },
  118. "url": "https://twitter.com/aaronpk/status/688518372170977280#favorited-by-16467582"
  119. }
  120. ],
  121. "comment": [
  122. {
  123. "type": "cite",
  124. "author": {
  125. "type": "card",
  126. "name": "Poetica",
  127. "photo": "https://s3-us-west-2.amazonaws.com/aaronparecki.com/twitter.com/192664bb706b2998ed42a50a860490b6aa1bb4926b458ba293b4578af599aa6f.png",
  128. "url": "http://poetica.com/"
  129. },
  130. "url": "https://twitter.com/poetica/status/689045331426803712",
  131. "published": "2016-01-18T03:23:03-08:00",
  132. "content": {
  133. "text": "@aaronpk @mozillapersona thanks very much! :)"
  134. }
  135. }
  136. ]
  137. }
  138. }
  139. ```
  140. #### Other Properties
  141. Other properties are returned in the response at the same level as the `data` property.
  142. * `url` - The effective URL that the document was retrieved from. This will be the final URL after following any redirects.
  143. * `code` - The HTTP response code returned by the URL. Typically this will be 200, but if the URL returned an alternate HTTP code that also included an h-entry (such as a 410 deleted notice with a stub h-entry), you can use this to find out that the original URL was actually deleted.
  144. ## Token API
  145. When verifying [Private Webmentions](https://indieweb.org/Private-Webmention#How_to_Receive_Private_Webmentions), you will need to exchange a code for an access token at the token endpoint specified by the source URL.
  146. XRay provides an API that will do this in one step. You can provide the source URL and code you got from the webmention, and XRay will discover the token endpoint, and then return you an access token.
  147. ```
  148. POST /token
  149. source=http://example.com/private-post
  150. &code=1234567812345678
  151. ```
  152. The response will be the response from the token endpoint, which will include an `access_token` property, and possibly an `expires_in` property.
  153. ```
  154. {
  155. "access_token": "eyJ0eXAXBlIjoI6Imh0dHB8idGFyZ2V0IjoraW0uZGV2bb-ZO6MV-DIqbUn_3LZs",
  156. "token_type": "bearer",
  157. "expires_in": 3600
  158. }
  159. ```
  160. If there was a problem fetching the access token, you will get one of the errors below in addition to the HTTP related errors returned by the parse API:
  161. * `no_token_endpoint` - Unable to find an HTTP header specifying the token endpoint.
  162. ## Installation
  163. ### From Source
  164. ```
  165. # Clone this repository
  166. git clone git@github.com:aaronpk/XRay.git
  167. cd XRay
  168. # Install dependencies
  169. composer install
  170. ```
  171. ### From Zip Archive
  172. * Download the latest release from https://github.com/aaronpk/XRay/releases
  173. * Extract to a folder on your web server
  174. ### Web Server Configuration
  175. Configure your web server to point to the `public` folder.
  176. Make sure all requests are routed to `index.php`. XRay ships with `.htaccess` files for Apache. For nginx, you'll need a rule like the following in your server config block.
  177. ```
  178. try_files $uri /index.php?$args;
  179. ```