Diferenzas

Isto amosa as diferenzas entre a revisión seleccionada e a versión actual da páxina.

--- composit:wsc09 [2014/11/10 10:54] – [Table] pablo.rodriguez.mier
+++ composit:wsc09 [2014/11/20 17:19] (actual) – [Evaluation] pablo.rodriguez.mier
@@ Liña 17: / Liña 17: @@
 ===== Purpose of this web document =====
 The purpose of this document is to extend the results obtained using the Web Service Challenge 2008 datasets
-with the most recent version of this challenge (Web Service Challenge 2009-2010). The main reason behind including only the results with the WSC'08
+with the most recent version of this challenge (Web Service Challenge 2009-2010). The main reason behind not including these results in the paper
-datasets is that the WSC'09-10 competition is focused on Quality-Of-Service (QoS) optimisation, and therefore better results can be achieved by developing concrete techniques
+is that the WSC'09-10 competition is focused on Quality-Of-Service (QoS) optimisation, and therefore better results can be achieved by developing concrete techniques
 for filtering services by their QoS. Moreover, this difference makes our results not comparable with the other approaches since we are providing semantic compositions
-just optimising the composition length and the number of services ignoring QoS.
+just optimising the composition length and the number of services ignoring QoS. We are currently working towards extending the proposed framework with QoS. Meanwhile, we think these results can be also interesting to compare the scalability of different
-We are currently working towards extending the proposed framework with QoS. Meanwhile, we think these results can be also interesting to compare the scalability of different
 composition frameworks.
@@ Liña 39: / Liña 37: @@
 ===== Evaluation =====
+We tested different configurations to study their individual performance and the overall impact on composition response times. In particular, we used the following configurations:
+  - **SPARQL D/M**: pure //SPARQL// Discovery / Matchmaking where all interactions with the Service and Knowledge Base managers are directly implemented as //SPARQL// queries. This is the typical approach of discovery engines and was the original implementation of iServe.
+  - **Index. D/SPARQL+Cache M**: I/O service discovery is based on an index. We additionally used herein an intermediate cache at the level of the concept matcher in order to avoid issuing recurrent //SPARQL// queries.
+  - **Full Indexed D/M**: both service discovery and concept matchmaking relied on local indexes pre-populated at load time (and updated with writes). In this configuration, service discovery and concept matchmaking do not need to issue any //SPARQL// query to the backed.
+The forward graph generation time + optimizations (**G. time**) and the total number of SPARQL queries generated (**#SPARQL**) are shown for each of these configurations. Column
+"**Composition**" shows the graph size (**G. size (opt)**, measured as the number of services) after the optimizations, and the total composition time (**Comp. time**) of the optimal service composition search. The last column "**Sol. (serv./length)**" shows the size of the optimal solution found (services, length).
+^  Dataset          ^  Graph size  ^  Discovery/Matchmaking (D/M)                                                                                        |||||^  Composition                      |^  Sol. (serv./length)  ^
+| :::               | :::          ^  1) SPARQL D/M                           |^  2) Index. D/SPARQL+Cache M             |^  3) Full Indexed D/M             || :::                               || :::                   |
+| :::               | :::          ^  G. time (s)                  ^  #SPARQL  ^  G. time (s)                 ^  #SPARQL  ^  G. time (s)          ^  #SPARQL  ^  G. size (opt)  ^  Comp. time (s)  | :::                   |
+|  WSC'09/10 - D01  |  49          |  96.83                        |  11454    |  28.23                       |  5217     |  0.23                 |  0        |  28             |  0.06            |  5 / 3                |
+|  WSC'09/10 - D02  |  93          |  720.10                       |  38614    |  92.71                       |  17663    |  0.43                 |  0        |  43             |  0.03            |  20 / 6               |
+|  WSC'09/10 - D03  |  65          |  609.00                       |  18410    |  67.98                       |  12130    |  0.60                 |  0        |  13             |  0.01            |  10 / 3               |
+|  WSC'09/10 - D04  |  199         |  2830.52                      |  120498   |  256.26                      |  47212    |  1.16                 |  0        |  118            |  >300            |  40 / 5               |
+|  WSC'09/10 - D05  |  216         |  5472.00                      |  181179   |  437.38                      |  78018    |  4.63                 |  0        |  96             |  0.13            |  30 / 19              |
+All datasets were solved with optimal values for composition length and number of services, showing a similar scalability as observed in the WSC'08 datasets (see graph below).
+{{:composit:wsc09-tsc-results.png?nolink|}}
-^  Dataset         ^  Graph size  ^  Discovery/Matchmaking (D/M)                                                                                        |||||^  Composition                      |^  Sol. (serv./length)  ^
-| :::              | :::          ^  1) SPARQL D/M                           |^  2) Index. D/SPARQL+Cache M             |^  3) Full Indexed D/M             || :::                               || :::                   |
-| :::              | :::          ^  G. time (s)                  ^  #SPARQL  ^  G. time (s)                 ^  #SPARQL  ^  G. time (s)          ^  #SPARQL  ^  G. size (opt)  ^  Comp. time (s)  | :::                   |
-| WSC'09/10 - D01  |  49          |  96.83                        |  11454    |  28.23                       |  5217     |  0.23                 |  0        |  28             |  0.06            |  5 / 3                |