{"id":5227,"date":"2016-06-22T07:30:47","date_gmt":"2016-06-22T07:30:47","guid":{"rendered":"https:\/\/modeling-languages.com\/?p=5227"},"modified":"2018-05-19T14:25:44","modified_gmt":"2018-05-19T14:25:44","slug":"json-schema-discoverer","status":"publish","type":"post","link":"https:\/\/modeling-languages.com\/json-schema-discoverer\/","title":{"rendered":"JSONDiscoverer: Visualizing the schema lurking behind JSON documents"},"content":{"rendered":"<p>Today we present the new version of our <a href=\"http:\/\/som-research.uoc.edu\/tools\/jsonDiscoverer\/#\/\" target=\"_blank\" rel=\"noopener\">JSON Discoverer<\/a> tool, which helps to discover the schema out of JSON documents. In short, given a set of schemaless JSON documents, our tool will parse them to infer and make explicit the implicit schema shared by all documents. This can be specially useful to learn how to work with data storedin a NoSQL backend or behind an API as happened with this happy user:<\/p>\n<blockquote class=\"twitter-tweet\" data-width=\"550\" data-dnt=\"true\">\n<p lang=\"en\" dir=\"ltr\"><a href=\"https:\/\/twitter.com\/softmodeling?ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">@softmodeling<\/a> <a href=\"https:\/\/twitter.com\/hashtag\/JSON?src=hash&amp;ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">#JSON<\/a> schema discoverer, layout of diagrams UML sometimes hidden under object diagram with large schema. Saved <a href=\"https:\/\/twitter.com\/hashtag\/Hours?src=hash&amp;ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">#Hours<\/a> <a href=\"https:\/\/twitter.com\/hashtag\/thanks?src=hash&amp;ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">#thanks<\/a><\/p>\n<p>&mdash; George Davies (@seivadge) <a href=\"https:\/\/twitter.com\/seivadge\/status\/744905925933281280?ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">June 20, 2016<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>In this blog post, we present a recap of the tool features and goals and include at the end a longer summary coming from our latest publication on this topic, titled\u00a0<em><a href=\"http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950705116300119\" target=\"_blank\" rel=\"noopener\">JSONDiscoverer:\u00a0Visualizing the schema lurking behind JSON documents<\/a>\u00a0<\/em>\u00a0to appear in the\u00a0<em>Knowledge-Based Systems<\/em> journal.<\/p>\n<p>In the last years the JavaScript Object Notation (<a href=\"http:\/\/json.org\" target=\"_blank\" rel=\"noopener\">JSON<\/a>) has been gaining in popularity since it provides a lightweigth data exchange format with a significant performance improvement.\u00a0JSON consists in sets of Object described by name\/value pairs. It is <strong>schemaless<\/strong>, i.e., there is no a structural definition of JSON objects, instead it is implicit. Schemaless data is particularly <a href=\"http:\/\/martinfowler.com\/articles\/schemaless\" target=\"_blank\" rel=\"noopener\">interesting in cases dealing with non-uniform data or in schema migration<\/a>, however, it can become a burden in data integration scenarios (e.g., consuming JSONbased APIs) where it becomes necessary to discover at least partially the underlying structure in order to properly process the data.<\/p>\n<p>Nowadays, a considerable number of web applications provide an external API consisting in a set of JSON-based services where all services are interrelated. Indeed, each service gives access to a subset of the application domain and developers must combine them to build any kind of non-trivial functionality on top of that API. Since JSON data is a schemaless format, deducing the right way of combining those services is not a trivial task. \u00a0<strong>JSONDiscoverer<\/strong> pretends to liberate developers from performing these tasks by inferring and visualizing the implicit schema of JSON data as well as the possible composition links among JSON-based Web APIs.<\/p>\n<p>The figure below\u00a0illustrates the typical development scenario where JSON-based Web APIs define a set of services, each one returning JSON documents when they are called.<\/p>\n<p><a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/doc-example.png\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-5272 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/doc-example.png\" alt=\"doc-example\" width=\"540\" height=\"408\" srcset=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/doc-example.png 800w, https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/doc-example-300x227.png 300w, https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/doc-example-65x50.png 65w\" sizes=\"(max-width: 540px) 100vw, 540px\" \/><\/a><\/p>\n<p>Our tool applies a discovery process to uncover the data model (i.e., schema) behind JSON-based Web APIs and assist on the discovery of composition links among them. These are currently the main functionalities provided:<\/p>\n<ul>\n<li><strong><a href=\"http:\/\/som-research.uoc.edu\/tools\/jsonDiscoverer\/#\/simple\" target=\"_blank\" rel=\"noopener\">Simple discovery<\/a><\/strong>, which discovers the schema of a given set of JSON documents returned by a single service.\u00a0You can find a more detailed description of this step\u00a0<a href=\"http:\/\/hal.inria.fr\/hal-00818945\" target=\"_blank\" rel=\"noopener\">our paper<\/a>\u00a0and the corresponding <a href=\"http:\/\/es.slideshare.net\/zirrus\/discovering-implicit-schemas-in-json-data\" target=\"_blank\" rel=\"noopener\">presentation slides<\/a>.<\/li>\n<li><strong><a href=\"http:\/\/som-research.uoc.edu\/tools\/jsonDiscoverer\/#\/advanced\" target=\"_blank\" rel=\"noopener\">Advanced discovery<\/a><\/strong>, which discovers the schema from a set of JSON-based services. First, the schema of each JSON-based service is discovered (by using the simple discoverer), then the resulting schemas are composed to obtain a general one.\u00a0You can find a more detailed description of this step\u00a0<a href=\"http:\/\/hal.inria.fr\/hal-00818945\" target=\"_blank\" rel=\"noopener\">our paper<\/a>\u00a0and the corresponding <a href=\"http:\/\/es.slideshare.net\/zirrus\/discovering-implicit-schemas-in-json-data\" target=\"_blank\" rel=\"noopener\">presentation slides<\/a>.<\/li>\n<li><strong><a href=\"http:\/\/som-research.uoc.edu\/tools\/jsonDiscoverer\/#\/composer\" target=\"_blank\" rel=\"noopener\">API Composer<\/a><\/strong>, which takes a set of API schemas, looks for composition links (i.e., common concepts or attributes) and generates a composition graph. The result is used to help developers to compose APIs. The tool currently incoporates a sequence diagram generator to visualize API compositions. This <a href=\"http:\/\/hal.inria.fr\/docs\/00\/97\/49\/38\/PDF\/29.pdf\" target=\"_blank\" rel=\"noopener\">paper<\/a> describes the process in detail (and these are the <a href=\"http:\/\/es.slideshare.net\/zirrus\/icwe2014-canovas-cabot\" target=\"_blank\" rel=\"noopener\">presentation slides<\/a>).<\/li>\n<\/ul>\n<p>Our tool draws schema information as UML class diagrams, including concepts (i.e., classes) and their properties (i.e., attributes and associations linking the different concepts). As the tools leverages <a href=\"https:\/\/www.eclipse.org\/modeling\/emf\/\" target=\"_blank\" rel=\"noopener\">EMF<\/a> (Eclipse Modeling Framework) to represent the schema information, Ecore models can also be obtained out of your JSON documents. Potential API compositions are represented by means of UML sequence diagrams showing the possible sequence of API calls.<\/p>\n<p>The tool is <a href=\"https:\/\/github.com\/SOM-Research\/jsonDiscoverer\/\" target=\"_blank\" rel=\"noopener\">available in GitHub<\/a>, where you will find the sources to import the tool in your application. We have also developed a <a href=\"http:\/\/som-research.uoc.edu\/tools\/jsonDiscoverer\/#\/\" target=\"_blank\" rel=\"noopener\">website<\/a> to use the tool in any web browser.<\/p>\n<p><a href=\"http:\/\/som-research.uoc.edu\/tools\/jsonDiscoverer\/#\/\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-5273 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/website.png\" alt=\"website\" width=\"551\" height=\"507\" srcset=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/website.png 1006w, https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/05\/website-300x276.png 300w\" sizes=\"(max-width: 551px) 100vw, 551px\" \/><\/a><\/p>\n<p>The following demonstration video shows how to use the website:<\/p>\n<p><center><br \/>\n<iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/FifSRxWHjIU\" width=\"420\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/center>This tool has been part of one of our resarch lines and\u00a0several papers have been published in that\u00a0regard. The <strong>last paper<\/strong> we have published describes the last version of the tool incorporating the previously described features. It is available <a href=\"http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950705116300119\" target=\"_blank\" rel=\"noopener\">here<\/a> but you can have a look below.<\/p>\n<p><strong>Abstract<\/strong><\/p>\n<p>The so-called API economy is pushing more and more companies to provide open Web APIs to access their data, typically using the JavaScript Object Notation (JSON) as interchange data format. While JSON has been designed to be easy to read and parse, their structure is implicit. This poses a serious problem when consuming and integrating Web APIs from different sources since it forces us to manually analyze each individual API in detail. This paper presents JSONDiscoverer, a tool that alleviates this problem by discovering (and visualizing) the implicit schema of JSON documents as well as possible composition links among JSON-based Web APIs.<\/p>\n<h2>1\u00a0 Introduction<\/h2>\n<p>The number of Web APIs is growing every day (the website <a href=\"http:\/\/www.programmableweb.com\" target=\"_blank\" rel=\"noopener\">Programmable web<\/a>\u00a0alone indexes over 13,000 APIs.)\u00a0opening the door to an unlimited number of new services built on top of such APIs. Most of those Web APIs use JavaScript Object Notation (JSON) as a data interchange format. JSON mimics the JavaScript syntax, thus becoming human readable and easily parseable. However, it is schemaless, i.e., JSON documents do not include an explicit definition of the structure of the JSON objects contained in them.<\/p>\n<p>This has several advantages but it is a serious problem when developing Web services that need to consume and exchange information among a set of APIs since developers need to figure out the structure of the JSON data provided by each one and the possible relationships between them. JSONDiscoverer pretends to liberate developers from performing these tasks by inferring and visualizing the implicit schema of JSON data as well as the possible composition links among JSON-based Web APIs. The tool has been made available as a web application. Since its release, JSONDiscoverer has been used to parse on average 375 JSON documents each month.<\/p>\n<h2>2\u00a0 Problem and Background<\/h2>\n<p>The first thing needed to reuse\/combine Web APIs is a good understanding of the data model behind them: what the data is about, what attributes each data object has, how they are related and so on.<\/p>\n<p>JSON being schemaless combined with the fact that the few languages to specify Web APIs are still under development (e.g., <a href=\"http:\/\/raml.org\" target=\"_blank\" rel=\"noopener\">RAML<\/a>\u00a0or <a href=\"https:\/\/swagger.io\/\" target=\"_blank\" rel=\"noopener\">Swagger<\/a>, which allow specifying API services and their parameters but not the full API schema) or are not widely used (e.g., JSON Schema\u00a0[1]) implies that developers must manually test Web APIs and try to deduce the data model lurking behind their services. Earlier research efforts (e.g., [2] and [3]) could be applied to analyze\u00a0JSON documents, however, they are specially tailored to NoSQL databases and do not provide assistance to integrate Web APIs.<\/p>\n<p>Figure 1 (grey boxes) illustrates the typical scenario when trying to integrate several JSON-based Web APIs. First, developers test each service provided and reverse engineer the implicit structure of the JSON data returned when calling them. Then, these individual service schemas need to be combined to build the full Web API schema (i.e., the global data model the API is giving access to). If two or more Web APIs need to be integrated, a last step is required aimed at identifying possible connections by searching for similar JSON elements that could be representing the same concept. Besides, the signature of each individual Web API service must be also considered to guarantee the accessibility of the target resources. Needless to say this is a time-consuming and error-prone task.<\/p>\n<p><a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/problem2.png\"><img loading=\"lazy\" decoding=\"async\" class=\" size-full wp-image-5229 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/problem2.png\" alt=\"problem2\" width=\"650\" height=\"240\" srcset=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/problem2.png 650w, https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/problem2-300x111.png 300w\" sizes=\"(max-width: 650px) 100vw, 650px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: center;\">Figure 1: Discovery of the implicit structure from a set of JSON-based Web APIs. The main functionalities of our tool are represented with black-filled rounded boxes while input\/output data is depicted as white-filled boxes.<\/p>\n<h2>3\u00a0 Software Functionalities<\/h2>\n<p>JSONDiscoverer alleviates this situation by executing an automatic discovery process on the JSON data that uncovers the schema behind JSON-based Web APIs and suggests possible composition paths.\u00a0The tool provides three main functionalities, which can be used separately or chained (see black-filled boxes in Figure 1):<\/p>\n<ol>\n<li><strong>Simple discovery<\/strong>. It discovers the schema of a given set of JSON documents returned by a single service. The more the better, since some properties of the (implicit) model can only be deduced when having several examples.<\/li>\n<li><strong>Advanced discovery<\/strong>. It takes the output of the simple discoverer for each service of a given Web API to infer its global schema.<\/li>\n<li><strong>Composer<\/strong>. It takes a set of inferred Web API schemas and looks for composition links (i.e., common concepts or attributes). As a result, it generates a composition graph. The <em>Composition Assistant<\/em> will use this graph to help you find composition paths among the Web APIs.<\/li>\n<\/ol>\n<p>JSONDiscoverer draws schema information as UML class diagrams, including concepts (i.e., classes) and their properties (i.e., attributes and associations linking the different concepts). Potential Web API compositions are represented by means of UML sequence diagrams showing the possible sequence of Web API calls.<\/p>\n<p>More information about the discovery rules applied in the simple and advanced discoverers can be found in [4], while a description about the composition rules and sequence diagram generation is done in [5].<\/p>\n<h2>4\u00a0 Implementation<\/h2>\n<p>JSONDiscoverer has been developed as a servlet-based web application including: (1) a backend developed in Java and providing the functionalities listed above; and (2) a front-end website implemented as an AngularJS web application. The website includes overlays to help newcomers to use the tool and a section explaining the inner workings for advanced users. Beyond this web frontend, JSONDiscoverer can also be executed from regular Java programs (see details on the tool website).<br \/>\nThe parsing and management of JSON documents is performed by using the <a href=\"https:\/\/github.com\/google\/gson\" target=\"_blank\" rel=\"noopener\">GSON library<\/a>. Model management relies on the <a href=\"https:\/\/eclipse.org\/modeling\/emf\" target=\"_blank\" rel=\"noopener\">Eclipse Modeling Framework (EMF)<\/a>\u00a0while we use <a href=\"https:\/\/marketplace.eclipse.org\/content\/emf-graphviz-emf2gv\" target=\"_blank\" rel=\"noopener\">EMF2GV<\/a>\u00a0for their rendering.<\/p>\n<h2>5\u00a0 Example<\/h2>\n<p>Figure 2 shows the Simple Discoverer page with an example. Once the user provides the JSON document to analyze in the textbox (or uses the default example for testing purposes) the button <em>Discover Schema<\/em> launches the discovery process, which sends the JSON document to the backend. As a result, the backend returns the domain model (i.e., schema) and JSON data (as an instance of the generated model), both as EMF models and pictures. The EMF models can be downloaded by the user and directly imported into other modeling tools for further analysis and manipulation.<\/p>\n<p style=\"text-align: center;\"><a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/screenshot2-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-5230\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/screenshot2-2.png\" alt=\"screenshot2-2\" width=\"800\" height=\"990\" srcset=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/screenshot2-2.png 800w, https:\/\/modeling-languages.com\/wp-content\/uploads\/2016\/04\/screenshot2-2-242x300.png 242w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/><\/a><\/p>\n<p style=\"text-align: center;\">Figure 2: Webpage for the simple discovery.<\/p>\n<h2>6\u00a0 Conclusion<\/h2>\n<p>In this paper we have presented JSONDiscoverer, a tool aimed at promoting the integration and composition of JSON-based Web APIs. As further work we plan to enhance the concept-matching heuristics in the (advanced) discovery and composition process, and to provide code-generation facilities to realize the selected API\u2019s integrations.<\/p>\n<h2>References<\/h2>\n<p>[1] IETF, JSON Schema Specification. http:\/\/json-schema.org\/.<\/p>\n<p>[2] M.\u00a0Klettke, U.\u00a0St\u00f6rl, S.\u00a0Scherzinger, Schema Extraction and Structural Outlier Detection for JSON-based NoSQL Data Stores, in: BTW conf., 2015, pp. 425\u2013444.<\/p>\n<p>[3] D.\u00a0Sevilla, S.\u00a0Feliciano, J.\u00a0G. Molina, Inferring Versioned Schemas from NoSQL Databases and Its Applications, in: ER conf., 2015, pp. 467\u2013480.<\/p>\n<p>[4] J.\u00a0L. C\u00e1novas Izquierdo, J.\u00a0Cabot, Discovering Implicit Schemas in JSON Data, in: ICWE conf., Vol. 7977, LNCS, 2013, pp. 68\u201383.<\/p>\n<p>[5] J.\u00a0L. C\u00e1novas Izquierdo, J.\u00a0Cabot, Composing JSON-based Web APIs, in: ICWE conf., Vol. 8541, LNCS, 2014, pp. 390\u2013399.<\/p>\n<span class=\"et_bloom_bottom_trigger\"><\/span>","protected":false},"excerpt":{"rendered":"<p>New version of our tool able to infer the shared schema among a set of schemaless JSON Documents<\/p>\n","protected":false},"author":5,"featured_media":5232,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[47,412,81,14],"tags":[95,101,438,433],"hashtags":[],"_links":{"self":[{"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/posts\/5227"}],"collection":[{"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/comments?post=5227"}],"version-history":[{"count":0,"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/posts\/5227\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/media\/5232"}],"wp:attachment":[{"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/media?parent=5227"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/categories?post=5227"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/tags?post=5227"},{"taxonomy":"hashtags","embeddable":true,"href":"https:\/\/modeling-languages.com\/wp-json\/wp\/v2\/hashtags?post=5227"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}