Optsicom Experiment Analyzer¶

This tool is aimed to perform analysis from data gathered during experiment execution. Because this application is tightly dependant on the format in wich data is recorded, also contain the libraries to save and retrieve experiemnt data.

Current status¶

Optsicom Experiment Analyzer currently is only a few of Java libraries that allows the following functionalities:

• Record experiment execution: This covers all features related to save execution data on disk for later analyses.
  • The values recorded during execution are saved in a file for each instance solved by a the methods configured in the experiment.
  • The files have a serialized Java format.
  • Each experiment is recorded in a folder.
  • There are tools (in form of Java libraries) to fusion several folders.
  • There is no metadata associated to an experiment as a whole, only a bunch of files on a folder.
• Analysis of recorded data: In algorithm research (especially in metaheuristic research) the data is analyzed using tables with common row and columns. Optsicom Experiment Analyzer has the following functionalities is this issue:
  • It is possible create tables in different ways. The rows and columns of the tables are higtly configurable.
  • The raw data (obtained from execution) can be processed in several ways:
    • The data can be processed by algoritm executed with each instance. Por example, a table can be generated with value obtained with each algorithm in each instance.
    • Also, it's possible to perform calculations considering all algorithms executed with the same instance. For example, some times the absolute value obtained is not important, is most important the ranking of an algorithm compared with other algorithms in the same experiment.
  • In order to synthesize data from a experiment with several instances, it possible to summarize data from each algorithm on all instances.
  • It is possible also to filter some algorithms or methods while creating a table.

Future improvements¶

The main functionality is almost complete. With the tool (in form of Java library) it can be done virtualy all needed tables that can be appear in a research paper. Nevertheless, there are several problems with this tool that need to be addressed:

• Record experiment execution:
  • It is usual that experimentation was executed in a different machine than development (This allows execute experiments in a controlled environment). Using files is the worst way to manage results because is needed perform a transfer to development machine. It is a big problem also when uses several machines to perform the execution. It is needed a more "distributed-friendly" way to store experiment data and database seems to be the best option. Nevertheless the developers need a easy way to save experiment data in local computer when developing algorithms and performing fine-tunning. In this case, an embebed database like Derby can be a good option.
  • On the other hand, a propietary Java binary format isn't the best way to save data that can be shared with other researchers. In this case, we need to complement the database with a textual based format to save data for experiments. XML seems the best option here.
  • Currently there are almost-none metadata associated with each experiment. Only the name of the folder (wrote by reseracher) is a hint to the experiment saved in the folder. We need a more powerfull mechanism to store metadata associated to an experiment. This allows us to search experiments, perform complex tables and reports, etc. For example, currently the problem solved by the experiment is not saved in the experiment and the analysed tool is unaware of maximization or minimization nature of the problem.
• Analysis of recorded data:
  • Currently each analysis must be executed explicitly. It gives control to the user, because you can configure each analysis as you want. But in the end, it is a lot time consuming because you are repeating the same analysis again and again depending on experiment and problem type. We need a way to perform a full analysis depending on the type of experiment (instances, algorithms, etc) and to generate a report containing all revelant information about the experiment. If the user need a specific table, it can configure and create it using Java code.
  • We need an UI to perform searches, configuring analysis, etc. Now all of these stuff is made with Java code and this is time consuming to configure all analysis with Java code. Nevertheless, this a very time consuming task, and with a reasonable automatic reporting when executing a typical experiment, it can be low priority.
  • We need an UI to control the experiment during execution. These UI is most related to experiment execution, not data analysis, but in certain sense, we want to perform data analysis "in real time" in order to evaluate an predict the behavior of designed algorithms.
  • It can be very usefull to have a mechanism to fine-grained controlling the data recorded. Currently, the only way to save data during executiong the code is calling a method in code. This is ok for user developed methods/algorithms, but is unaceptable for library methods because the user has no control over these provided methods. We need a way to activate or deactivate "traces" in algorithms when configuring the experiment. If an trace is deactivated, no overhead should be paid in execution. It is possible that we need some sort of bytecode manipulation/instrumentation to obtain this requirements.