≪ View Page

Diferenzas

Isto amosa as diferenzas entre a revisión seleccionada e a versión actual da páxina.

Enlazar a esta vista de comparación

Both sides previous revisionPrevious revision
Next revision		Previous revision
inv:composit:validation [2013/02/28 10:22] – [Evaluation] pablo.rodriguez.mierinv:composit:validation [2013/02/28 18:31] (actual) – pablo.rodriguez.mier
Liña 1: Liña 1:
-====== ComposIT: experiments and comparison with AI planners ======+====== ComposIT: A Semantic Web Service Composition Engine ====== 
 +**Authors: Pablo Rodriguez-Mier, Carlos Pedrinaci, Manuel Lama, and Manuel Mucientes**
  
-In previous works [1][2], ComposIT was analyzed and compared with the winners of the Web Service Challenge 2008 (WSC'08). These experiments demonstrated that ComposIT can obtain optimal solutions, outperforming the results obtained by the winners with respect the number of services and the composition length. However, the aim of the experiments presented here is to compare the performance of ComposIT with two open source state-of-the-art composition engines, OWLS-Xplan and PORSCE-II, which use classical AI planning algorithms. + 
 +//Abstract//—One major advantage of web services is the ability to be combined to create composite services on-demand 
 +through automatic composition techniques. However, although the inclusion of semantics allows a greater precision in the 
 +description of their functionality, and therefore greater composition capabilities, the current application of Semantic Web Services 
 +(SWS) composition techniques still remains limited due in part to both lack of publicly available, robust, and scalable software 
 +composition engines and the heterogeneity that exists among the different SWS description languages. In this paper we introduce 
 +ComposIT, a fast and scalable composition engine which is able to automatically compose multiple heterogeneous services from 
 +the point of view of the semantic input-output matching thanks to the use of a minimal service model. We also present a complete 
 +analysis of two publicly available state-of-the-art composition planners and a comparison between ComposIT and these planners. 
 +To carry out this task, we developed a benchmarking tool that automates the evaluation process of the different composition 
 +algorithms using an adapted version of the datasets from the Web Service Challenge 2008. Results obtained demonstrate that 
 +ComposIT outperforms classical planners both in terms of scalability and performance. 
 + 
 + 
 +===== Purpose of this Web ===== 
 + 
 +In previous works((P. Rodriguez-Mier, M. Mucientes, and M. Lama, “Automatic Web Service Composition with a Heuristic-Based Search Algorithm,” in //International Conference on Web Services (ICWS)//. IEEE Computer Society, 2011, pp. 81–88.)) ((P. Rodriguez-Mier, M. Mucientes, J. Vidal, and M. Lama, “An optimal and complete algorithm for automatic web service composition,” //International Journal of Web Services Research (IJWSR)//, vol. 9, no. 2, pp. 1–20, 2012.)), ComposIT was analyzed and compared with the winners of the Web Service Challenge 2008 (WSC'08). These experiments demonstrated that ComposIT can obtain optimal solutions, outperforming the results obtained by the winners with respect to the number of services and the composition length. However, the aim of the experiments presented here is to compare the performance of ComposIT with two open source state-of-the-art composition engines, OWLS-Xplan and PORSCE-II, which use classical AI planning algorithms. 
  
 The experimentation is focused on the generation of semantically valid composite services taking into account the semantic information regarding the inputs and outputs of services in absence of preconditions and effects. To carry out this comparison, we developed a benchmarking tool that  The experimentation is focused on the generation of semantically valid composite services taking into account the semantic information regarding the inputs and outputs of services in absence of preconditions and effects. To carry out this comparison, we developed a benchmarking tool that 
-automates the evaluation process of the different composition algorithms using an adapted version of the WSC'08 datasets. This page extends the details of the evaluation done.+automates the evaluation process of the different composition algorithms using an adapted version of the WSC'08 datasets. **The purpose of this 
 +page is to extend the details of the evaluation explained in the original paper.**
  
 ===== Datasets ===== ===== Datasets =====
Liña 58: Liña 76:
  
  
-These tables show: the number of services of each dataset (column #Serv); the number of services of the optimal solution (column #Serv. Sol.); the length of the shortest solution (column #Length); the average number of inputs and outputs (column  Avg. In./Out.); the number of inputs provided by the user (column #Init. con.) and the number of wanted output concepts (column #Goal con.). The goal is to find, for each dataset, the optimal +Exact-Matching datasets were calculated by extending the outputs of each web service, including all superclasses of each output as an output of the service itself (semantic expansion). Thus, the average number of outputs is bigger than in the other datasets. The semantic expansion transforms a semantic matching problem into a exact matching problem, when exact and plug-in match is used to perform the semantic matchmaking. This allows us to test composition algorithms (that do not use semantic reasoners) with the WSC'08 datasets. For example, suppose that a service S1 provides the instance "my_car" which is an instance of the concept "sedan", whereas another service S2 requires an input "car". Given that "my_car" is a sedan "car" (plug-in match), S2 can use it as an input. This behavior can be simulated by adding superclasses of "my_car" to the service. Thus, service S1
-composite service (minimum number of services, minimum length) that satisfies the goal concepts, using only the initial inputs provided. +
- +
-Exact-Matching datasets were calculated by extending the outputs of each web service, including all superclasses of each output as an output of the service itself (semantic expansion). Thus, the average number of outputs is bigger than in the other datasets. The semantic expansion transforms a semantic matching problem into a exact matching problem, when exact and plug-in match is used to perform the semantic matchmaking. This allows us to test composition algorithms (that do not use semantic reasoners) with the WSC'08 datasets. For example, suppose that a service S1 provides the instance "my_car" which is an instance of the concept "sedan", whereas another service S2 requires an input "car". Given that "my_car" is a sedan "car" (plug-in match), S2 can use it as an input. This behavior can be simulated by adding superclasses of my_car to the service. Thus, service S1+
 will return outputs "my_car", "car", "sedan" and "vehicle" after the semantic expansion. will return outputs "my_car", "car", "sedan" and "vehicle" after the semantic expansion.
  
-The figure below represents an example of the semantic expansion. In the first case, a semantic reasoner is required to match output "my_car" to "car". In the second case, the service returns the output "my_car" as well as all superclasses (sedan, car, vehicle). In this case, the output "car" from S1 matches exactly the input "car" from S2.+The next figure represents an example of the semantic expansion. In the first case, a semantic reasoner is required to match output "my_car" to "car". In the second case, the service returns the output "my_car" as well as all superclasses (sedan, car, vehicle). In this case, the output "car" from S1 matches exactly the input "car" from S2.
  
 {{:inv:composit:match-example2.png?200|}} {{:inv:composit:match-example2.png?200|}}
Liña 127: Liña 142:
 === 2. Semantic-Matching evaluation results === === 2. Semantic-Matching evaluation results ===
  
-In this experiment, we evaluate the performance of ComposIT and PORSCE-II with the semantic-matching datasets. Note that OWLS-Xplan does not support semantic matchmaking. Therefore, this algorithm has been excluded from this evaluation and also from the WSC'08 evaluation, which requires semantic matchmaking. +Results of the semantic analysis done to detect the optimal values of the thresholds for PORSCE-II.
- +
-PORSCE-II uses different threshold values to configure the maximum semantic concept distance allowed for a valid semantic matching. +
-The values of these thresholds are directly related to the performance of the algorithm so that the higher the thresholds, the slower the algorithm runs, but the better the quality of the solutions. Given the difficulty of selecting an appropriate value for the threshold without compromising too much the performance, we selected the optimal value for each dataset based on a study using the Semantic-matching datasets in which we analyzed the performance of PORSCE-II by using incremental values of the threshold from 1 to 19, which corresponds with the maximum depth for all ontologies.+
  
 ^  Plug-in threshold  ^  Time (for Semantic-Matching 01)  ^  % Datasets solved  ^ Semantic-Matching datasets solved  ^  Performance  ^ ^  Plug-in threshold  ^  Time (for Semantic-Matching 01)  ^  % Datasets solved  ^ Semantic-Matching datasets solved  ^  Performance  ^
Liña 153: Liña 165:
 | 19                  | 58981.02                          | 60%                 | 01, 02, 03, 04, 06, 07             | 11.40%        | | 19                  | 58981.02                          | 60%                 | 01, 02, 03, 04, 06, 07             | 11.40%        |
  
- +Based 
-Figure below shows the percentage of semantic problems solved by PORSCE-II with different values of the threshold and the performance +on these results, we selected the following optimal threshold //N// for each datasets: //N=1// for //Semantic-Matching 01//, //N=2// for //Semantic-Matching 02// and //03//, //N=3// for //Semantic-Matching 04//, //N=4// for //Semantic-Matching 07// and //N=6// for //Semantic-Matching 06//.
-measured by comparing the time taken by the algorithm to solve the //Semantic-Matching 01// with the different thresholds. As can be seen, +
-the computational cost increases (performance decreases) when the threshold is incremented. Thus, using a threshold of 10, the algorithm +
-obtains a performance close to 20\%, which means that the algorithm is about 1/0.20=5 times slower than using a threshold of 1. Based +
-on these results, we selected the optimal thresholds //N// for each datasets: //N=1// for //Semantic-Matching 01//, //N=2// for //Semantic-Matching 02// and //03//, //N=3// for //Semantic-Matching 04//, //N=4// for //Semantic-Matching 07// and //N=6// for //Semantic-Matching 06//. Note that +
-ComposIT does not require any special configuration for the semantic datasets as it calculates matches at unlimited depth.  +
- +
-{{:inv:composit:porsce-thresholds.png?400|}} +
- +
  
 ^        Dataset        ^    Algorithm    ^      Solution      ^^                                        Execution (ms)                                        ^^^^^^^^ ^        Dataset        ^    Algorithm    ^      Solution      ^^                                        Execution (ms)                                        ^^^^^^^^
Liña 220: Liña 223:
 </code> </code>
 Where algorithm.jar is one of the available algorithms:  Where algorithm.jar is one of the available algorithms: 
-  * [[http://apps.citius.usc.es/upload/?a=d&i=M5CkkF|CompositAlgorithm.jar]] +  * ComposIT: CompositAlgorithm.jar ([[http://apps.citius.usc.es/upload/?a=d&i=M5CkkF|download]]) 
-  * [[http://apps.citius.usc.es/upload/?a=d&i=Wv1RNM|PorsceAlgorithm.jar]] +  * PORSCE-II: PorsceAlgorithm.jar ([[http://apps.citius.usc.es/upload/?a=d&i=Wv1RNM|download]]) 
-  * [[http://apps.citius.usc.es/upload/?a=d&i=Re6Nv2|OWLSXplanAlgorithm.jar]]+  * OWLS-Xplan: OWLSXplanAlgorithm.jar ([[http://apps.citius.usc.es/upload/?a=d&i=Re6Nv2|download]])
  
 +<note important>
 +These versions of the OWLS-Xplan and PORSCE-II were modified to support the integration with the test platform. Original versions
 +of these algorithms can be downloaded here:
 +  * PORSCE-II: http://www.dit.hua.gr/~raniah/research.html
 +  * OWLS-Xplan 2.0: http://projects.semwebcentral.org/projects/owls-xplan/
 +</note>
 You can launch also a background test from the command line, with the following syntax: You can launch also a background test from the command line, with the following syntax:
  
Liña 240: Liña 249:
 PORSCE-II: PORSCE-II:
 <code> <code>
-java -Xmx1024M -Xms512M -jar algorithm.jar "PORSCE-II" "...\Datasets\Exact-matching\01" 80 > 01.txt+java -Xmx1024M -Xms512M -jar PorsceAlgorithm.jar "PORSCE-II" "...\Datasets\Exact-matching\01" 80 > 01.txt
 </code> </code>