You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

538 lines
18 KiB

5 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
  1. XRay
  2. ====
  3. XRay parses structured content from a URL.
  4. ## Discovering Content
  5. XRay will parse content in the following formats. First the URL is checked against known services:
  6. * Instagram
  7. * Twitter
  8. * GitHub
  9. * XKCD
  10. * Hackernews
  11. * Facebook (public events)
  12. If the contents of the URL is XML or JSON, then XRay will parse the Atom, RSS or JSONFeed formats.
  13. Finally, XRay looks for Microformats on the page and will determine the content from that.
  14. * h-card
  15. * h-entry
  16. * h-event
  17. * h-review
  18. * h-recipe
  19. * h-product
  20. * h-item
  21. * h-feed
  22. ## Library
  23. XRay can be used as a library in your PHP project. The easiest way to install it and its dependencies is via composer.
  24. ```
  25. composer require p3k/xray
  26. ```
  27. You can also [download a release](https://github.com/aaronpk/XRay/releases) which is a zip file with all dependencies already installed.
  28. ### Parsing
  29. ```php
  30. $xray = new p3k\XRay();
  31. $parsed = $xray->parse('https://aaronparecki.com/2017/04/28/9/');
  32. ```
  33. If you already have an HTML or JSON document you want to parse, you can pass it as a string in the second parameter.
  34. ```php
  35. $xray = new p3k\XRay();
  36. $html = '<html>....</html>';
  37. $parsed = $xray->parse('https://aaronparecki.com/2017/04/28/9/', $html);
  38. ```
  39. ```php
  40. $xray = new p3k\XRay();
  41. $jsonfeed = '{"version":"https://jsonfeed.org/version/1","title":"Manton Reece", ... ';
  42. // Note that the JSON document must be passed in as a string in this case
  43. $parsed = $xray->parse('https://manton.micro.blog/feed.json', $jsonfeed);
  44. ```
  45. In both cases, you can add an additional parameter to configure various options of how XRay will behave. Below is a list of the options.
  46. * `timeout` - The timeout in seconds to wait for any HTTP requests
  47. * `max_redirects` - The maximum number of redirects to follow
  48. * `include_original` - Will also return the full document fetched
  49. * `target` - Specify a target URL, and XRay will first check if that URL is on the page, and only if it is, will continue to parse the page. This is useful when you're using XRay to verify an incoming webmention.
  50. * `expect=feed` - If you know the thing you are parsing is a feed, include this parameter which will avoid running the autodetection rules and will provide better results for some feeds.
  51. Additional parameters are supported when making requests that use the Twitter or GitHub API. See the Authentication section below for details.
  52. ```php
  53. $xray = new p3k\XRay();
  54. $parsed = $xray->parse('https://aaronparecki.com/2017/04/28/9/', [
  55. 'timeout' => 30
  56. ]);
  57. $parsed = $xray->parse('https://aaronparecki.com/2017/04/28/9/', $html, [
  58. 'target' => 'http://example.com/'
  59. ]);
  60. ```
  61. The `$parsed` return value will look like the below. See "Primary Data" below for an explanation of the vocabularies returned.
  62. ```
  63. $parsed = Array
  64. (
  65. [data] => Array
  66. (
  67. [type] => card
  68. [name] => Aaron Parecki
  69. [url] => https://aaronparecki.com/
  70. [photo] => https://aaronparecki.com/images/profile.jpg
  71. )
  72. [url] => https://aaronparecki.com/
  73. [code] => 200,
  74. [source-format] => mf2+html
  75. )
  76. ```
  77. ### Processing Microformats2 JSON
  78. If you already have a parsed Microformats2 document as an array, you can use a special function to process it into XRay's native format. Make sure you pass the entire parsed document, not just the single item.
  79. ```php
  80. $html = '<div class="h-entry"><p class="p-content p-name">Hello World</p><img src="/photo.jpg"></p></div>';
  81. $mf2 = Mf2\parse($html, 'http://example.com/entry');
  82. $xray = new p3k\XRay();
  83. $parsed = $xray->process('http://example.com/entry', $mf2); // note the use of `process` not `parse`
  84. Array
  85. (
  86. [data] => Array
  87. (
  88. [type] => entry
  89. [post-type] => photo
  90. [photo] => Array
  91. (
  92. [0] => http://example.com/photo.jpg
  93. )
  94. [content] => Array
  95. (
  96. [text] => Hello World
  97. )
  98. )
  99. [url] => http://example.com/entry
  100. [source-format] => mf2+json
  101. )
  102. ```
  103. ### Rels
  104. You can also use XRay to fetch all the rel values on a page, merging the list of HTTP `Link` headers with rel values with the HTML rel values on the page.
  105. ```php
  106. $xray = new p3k\XRay();
  107. $rels = $xray->rels('https://aaronparecki.com/');
  108. ```
  109. This will return a similar response to the parser, but instead of a `data` key containing the parsed page, there will be `rels`, an associative array. Each key will contain an array of all the values that match that rel value.
  110. ```
  111. Array
  112. (
  113. [url] => https://aaronparecki.com/
  114. [code] => 200
  115. [rels] => Array
  116. (
  117. [hub] => Array
  118. (
  119. [0] => https://switchboard.p3k.io/
  120. )
  121. [authorization_endpoint] => Array
  122. (
  123. [0] => https://aaronparecki.com/auth
  124. )
  125. ...
  126. ```
  127. ### Feed Discovery
  128. You can use XRay to discover the types of feeds available at a URL.
  129. ```php
  130. $xray = new p3k\XRay();
  131. $feeds = $xray->feeds('http://percolator.today');
  132. ```
  133. This will fetch the URL, check for a Microformats feed, as well as check for rel=alternates pointing to Atom, RSS or JSONFeed URLs. The response will look like the below.
  134. ```
  135. Array
  136. (
  137. [url] => https://percolator.today/
  138. [code] => 200
  139. [feeds] => Array
  140. (
  141. [0] => Array
  142. (
  143. [url] => https://percolator.today/
  144. [type] => microformats
  145. )
  146. [1] => Array
  147. (
  148. [url] => https://percolator.today/podcast.xml
  149. [type] => rss
  150. )
  151. )
  152. )
  153. ```
  154. ### Customizing the User Agent
  155. To set a unique user agent, (some websites will require a user agent be set), you can set the `http` property of the object to a `p3k\HTTP` object.
  156. ```php
  157. $xray = new p3k\XRay();
  158. $xray->http = new p3k\HTTP('MyProject/1.0.0 (http://example.com/)');
  159. $xray->parse('http://example.com/');
  160. ```
  161. ## API
  162. XRay can also be used as an API to provide its parsing capabilities over an HTTP service.
  163. To parse a page and return structured data for the contents of the page, simply pass a url to the `/parse` route.
  164. ```
  165. GET /parse?url=https://aaronparecki.com/2016/01/16/11/
  166. ```
  167. To conditionally parse the page after first checking if it contains a link to a target URL, also include the target URL as a parameter. This is useful when using XRay to verify an incoming webmention.
  168. ```
  169. GET /parse?url=https://aaronparecki.com/2016/01/16/11/&target=http://example.com
  170. ```
  171. In both cases, the response will be a JSON object containing a key of "type". If there was an error, "type" will be set to the string "error", otherwise it will refer to the kind of content that was found at the URL, most often "entry".
  172. You can also make a POST request with the same parameter names.
  173. If you already have an HTML or JSON document you want to parse, you can include that in the POST parameter `body`. This POST request would look like the below:
  174. ```
  175. POST /parse
  176. Content-type: application/x-www-form-urlencoded
  177. url=https://aaronparecki.com/2016/01/16/11/
  178. &body=<html>....</html>
  179. ```
  180. or for Twitter/GitHub/Facebook where you might have JSON,
  181. ```
  182. POST /parse
  183. Content-type: application/x-www-form-urlencoded
  184. url=https://github.com/aaronpk/XRay
  185. &body={"repo":......}
  186. ```
  187. ### Parameters
  188. XRay accepts the following parameters when calling `/parse`
  189. * `url` - the URL of the page to parse
  190. * `target` - Specify a target URL, and XRay will first check if that URL is on the page, and only if it is, will continue to parse the page. This is useful when you're using XRay to verify an incoming webmention.
  191. * `timeout` - The timeout in seconds to wait for any HTTP requests
  192. * `max_redirects` - The maximum number of redirects to follow
  193. * `include_original` - Will also return the full document fetched
  194. * `expect=feed` - If you know the thing you are parsing is a feed, include this parameter which will avoid running the autodetection rules and will provide better results for some feeds.
  195. ### Authentication
  196. If the URL you are fetching requires authentication, include the access token in the parameter "token", and it will be included in an "Authorization" header when fetching the URL. (It is recommended to use a POST request in this case, to avoid the access token potentially being logged as part of the query string.) This is useful for [Private Webmention](https://indieweb.org/Private-Webmention) verification.
  197. ```
  198. POST /parse
  199. url=https://aaronparecki.com/2016/01/16/11/
  200. &target=http://example.com
  201. &token=12341234123412341234
  202. ```
  203. ### API Authentication
  204. XRay uses the Twitter, Github and Facebook APIs to fetch posts, and those API require authentication. In order to keep XRay stateless, it is required that you pass in the credentials to the parse call.
  205. You should only send the credentials when the URL you are trying to parse is a Twitter URL, a GitHub URL or a Facebook URL, so you'll want to check for whether the hostname is `twitter.com`, `github.com`, etc. before you include credentials in this call.
  206. #### Twitter Authentication
  207. XRay uses the Twitter API to fetch Twitter URLs. You can register an application on the Twitter developer website, and generate an access token for your account without writing any code, and then use those credentials when making an API request to XRay.
  208. * `twitter_api_key` - Your application's API key
  209. * `twitter_api_secret` - Your application's API secret
  210. * `twitter_access_token` - Your Twitter access token
  211. * `twitter_access_token_secret` - Your Twitter secret access token
  212. #### GitHub Authentication
  213. XRay uses the GitHub API to fetch GitHub URLs, which provides higher rate limits when used with authentication. You can pass a GitHub access token along with the request and XRay will use it when making requests to the API.
  214. * `github_access_token` - A GitHub access token
  215. #### Facebook Authentication
  216. XRay uses the Facebook API to fetch Facebook URLs. You can create a Facebook App on Facebooks developer website.
  217. * facebook_app_id - Your application's App ID
  218. * facebook_app_secret - Your application's App Secret
  219. At this moment, XRay is able to get it's own access token from those credentials.
  220. ### Error Response
  221. ```json
  222. {
  223. "error": "not_found",
  224. "error_description": "The URL provided was not found"
  225. }
  226. ```
  227. Possible errors are listed below:
  228. * `not_found`: The URL provided was not found. (Returned 404 when fetching)
  229. * `ssl_cert_error`: There was an error validating the SSL certificate. This may happen if the SSL certificate has expired.
  230. * `ssl_unsupported_cipher`: The web server does not support any of the SSL ciphers known by the service.
  231. * `timeout`: The service timed out trying to connect to the URL.
  232. * `invalid_content`: The content at the URL was not valid. For example, providing a URL to an image will return this error.
  233. * `no_link_found`: The target link was not found on the page. When a target parameter is provided, this is the error that will be returned if the target could not be found on the page.
  234. * `no_content`: No usable content could be found at the given URL.
  235. * `unauthorized`: The URL returned HTTP 401 Unauthorized.
  236. * `forbidden`: The URL returned HTTP 403 Forbidden.
  237. ### Response Format
  238. ```json
  239. {
  240. "data":{
  241. "type":"entry",
  242. "post-type":"photo",
  243. "published":"2017-03-01T19:00:33-08:00",
  244. "url":"https://aaronparecki.com/2017/03/01/14/hwc",
  245. "category":[
  246. "indieweb",
  247. "hwc"
  248. ],
  249. "photo":[
  250. "https://aaronparecki.com/2017/03/01/14/photo.jpg"
  251. ],
  252. "syndication":[
  253. "https://twitter.com/aaronpk/status/837135519427395584"
  254. ],
  255. "content":{
  256. "text":"Hello from Homebrew Website Club PDX! Thanks to @DreamHost for hosting us! 🍕🎉 #indieweb",
  257. "html":"Hello from Homebrew Website Club PDX! Thanks to <a href=\"https://twitter.com/DreamHost\">@DreamHost</a> for hosting us! <a href=\"https://aaronparecki.com/emoji/%F0%9F%8D%95\">🍕</a><a href=\"https://aaronparecki.com/emoji/%F0%9F%8E%89\">🎉</a> <a href=\"https://aaronparecki.com/tag/indieweb\">#indieweb</a>"
  258. },
  259. "author":{
  260. "type":"card",
  261. "name":"Aaron Parecki",
  262. "url":"https://aaronparecki.com/",
  263. "photo":"https://aaronparecki.com/images/profile.jpg"
  264. }
  265. },
  266. "url":"https://aaronparecki.com/2017/03/01/14/hwc",
  267. "code":200,
  268. "source-format":"mf2+html"
  269. }
  270. ```
  271. #### Primary Data
  272. The primary object on the page is returned in the `data` property. This will indicate the type of object (e.g. `entry`), and will contain the vocabulary's properties that it was able to parse from the page.
  273. * `type` - the Microformats 2 vocabulary found for the primary object on the page, without the `h-` prefix (e.g. `entry`, `event`)
  274. * `post-type` - only for "posts" (e.g. not for `card`s) - the [Post Type](https://www.w3.org/TR/post-type-discovery/) of the post (e.g. (`note`, `photo`, `reply`))
  275. If a property supports multiple values, it will always be returned as an array. The following properties support multiple values:
  276. * `in-reply-to`
  277. * `like-of`
  278. * `repost-of`
  279. * `bookmark-of`
  280. * `follow-of`
  281. * `syndication`
  282. * `photo` (of an entry, not of a card)
  283. * `video`
  284. * `audio`
  285. * `category`
  286. The content will be an object that always contains a "text" property and may contain an "html" property if the source documented published HTML content. The "text" property must always be HTML escaped before displaying it as HTML, as it may include unescaped characters such as `<` and `>`.
  287. The author will always be set in the entry if available. The service follows the [authorship discovery](https://indieweb.org/authorship) algorithm to try to find the author information elsewhere on the page if it is not inside the entry in the source document.
  288. All URLs provided in the output are absolute URLs. If the source document contains a relative URL, it will be resolved first.
  289. #### Post Type Discovery
  290. XRay runs the [Post Type Discovery](https://www.w3.org/TR/post-type-discovery/) algorithm and also includes a `post-type` property.
  291. The following post types are returned, which are slightly expanded from what is currently documented by the Post Type Discovery spec.
  292. * `event`
  293. * `recipe`
  294. * `review`
  295. * `rsvp`
  296. * `repost`
  297. * `like`
  298. * `reply`
  299. * `bookmark`
  300. * `follow`
  301. * `checkin`
  302. * `video`
  303. * `audio`
  304. * `photo`
  305. * `article`
  306. * `note`
  307. #### Other Properties
  308. Other properties are returned in the response at the same level as the `data` property.
  309. * `url` - The effective URL that the document was retrieved from. This will be the final URL after following any redirects.
  310. * `code` - The HTTP response code returned by the URL. Typically this will be 200, but if the URL returned an alternate HTTP code that also included an h-entry (such as a 410 deleted notice with a stub h-entry), you can use this to find out that the original URL was actually deleted.
  311. * `source-format` - Indicates the format of the source URL that was used to generate the parsed result. Possible values are:
  312. * `mf2+html`
  313. * `mf2+json`
  314. * `feed+json`
  315. * `xml`
  316. * `instagram`/`facebook`/`github`/`xkcd`
  317. #### Feeds
  318. XRay can return information for several kinds of feeds. The URL (or body) passed to XRay will be checked for the following formats:
  319. * XML (Atom and RSS)
  320. * JSONFeed (https://jsonfeed.org)
  321. * Microformats [h-feed](https://indieweb.org/h-feed)
  322. If the page being parsed represents a feed, then the response will look like the following:
  323. ```json
  324. {
  325. "data": {
  326. "type": "feed",
  327. "items": [
  328. {...},
  329. {...}
  330. ]
  331. }
  332. }
  333. ```
  334. Each object in the `items` array will contain a parsed version of the item, in the same format that XRay normally returns. When parsing Microformats feeds, the [authorship discovery](https://indieweb.org/authorship) will be run for each item to build out the author info.
  335. Atom, RSS and JSONFeed will all be normalized to XRay's vocabulary, and only recognized properties will be returned.
  336. ## Rels API
  337. There is also an API method to parse and return all rel values on the page, including HTTP `Link` headers and HTML rel values.
  338. ```
  339. GET /rels?url=https://aaronparecki.com/
  340. ```
  341. See [above](#rels) for the response format.
  342. ## Feed Discovery API
  343. ```
  344. GET /feeds?url=https://aaronparecki.com/
  345. ```
  346. See [above](#feed-discovery) for the response format.
  347. ## Token API
  348. When verifying [Private Webmentions](https://indieweb.org/Private-Webmention#How_to_Receive_Private_Webmentions), you will need to exchange a code for an access token at the token endpoint specified by the source URL.
  349. XRay provides an API that will do this in one step. You can provide the source URL and code you got from the webmention, and XRay will discover the token endpoint, and then return you an access token.
  350. ```
  351. POST /token
  352. source=http://example.com/private-post
  353. &code=1234567812345678
  354. ```
  355. The response will be the response from the token endpoint, which will include an `access_token` property, and possibly an `expires_in` property.
  356. ```
  357. {
  358. "access_token": "eyJ0eXAXBlIjoI6Imh0dHB8idGFyZ2V0IjoraW0uZGV2bb-ZO6MV-DIqbUn_3LZs",
  359. "token_type": "bearer",
  360. "expires_in": 3600
  361. }
  362. ```
  363. If there was a problem fetching the access token, you will get one of the errors below in addition to the HTTP related errors returned by the parse API:
  364. * `no_token_endpoint` - Unable to find an HTTP header specifying the token endpoint.
  365. ## Installation
  366. ### From Source
  367. ```
  368. # Clone this repository
  369. git clone git@github.com:aaronpk/XRay.git
  370. cd XRay
  371. # Install dependencies
  372. composer install
  373. ```
  374. ### From Zip Archive
  375. * Download the latest release from https://github.com/aaronpk/XRay/releases
  376. * Extract to a folder on your web server
  377. ### Web Server Configuration
  378. Configure your web server to point to the `public` folder.
  379. Make sure all requests are routed to `index.php`. XRay ships with `.htaccess` files for Apache. For nginx, you'll need a rule like the following in your server config block.
  380. ```
  381. try_files $uri /index.php?$args;
  382. ```