Link to the Evaluation Interface
An evaluation webpage can be found at: http://bio-eval.scai.fraunhofer.de/cgi-bin/General_server.rc
Purpose and Set up of the Evaluation Interface
The interface lets participants experience the evaluation procedure that will be used in the official evaluation. All results are purely for the own feedback of participating teams and will not be officially evaluated in any way.
At present, you can evaluate BEL statements generated from the Sample set, under the assumption that the Training set is used for training a system, and the Sample set is used as a development set. Later on, we will also include the Training Set into the Evaluation Interface, so that cross validation experiments are possible.
How to use the Evaluation Interface
The submission has to be structured as one statement per line, each line composed by two elements, separated by a tab character (instead of a tab character any whitespace larger than 2 spaces is accepted):
1. sentence identifier
2. BEL statement or BEL fragment
SEN:10000046 cat(p(HGNC:F2R)) increases p(HGNC:IL6)
You can either copy and paste your statements in the text area (marked as "Input") or upload a file ("File Input"). The structure of the submission should be the same in both cases.
You should also select the level at which you want your statements to be evaluated (using the pull-down menu "Submission"):
1. "Term Level" (T)
2. "Function Level" (F)
3. "Relation Level" (R)
Evaluation on the Full Statement level (S) is included under the Relation Level.
Please also add an email address for confirmation of identity. Currently, the email addresses that you used for registration on biocreative.org are accepted.
Additionally the address "email@example.com" can be used for testing purposes (though we recommend using your own address). Please note that for legal reasons this address will be soon disabled and you will be required to use a registered e-mail address.
Please do not hesitate to ask if you wish any other addresses to be added to the system.
Note: The submission at any level will also be evaluated at the levels below.
So for example a submission at level 2 will be evaluated for the functions that it contains, but also for the terms that it contains (separate scores are provided). A submission at level 3 will be evaluated for terms (T), functions (F), the relation (R), and as a full BEL statement (S).
Additionally, the system provides secondary level evaluations for the function level (Fs) and relation level (Rs). The secondary level evaluation score for the term level (Ts) is not yet available.
For a detailed description of the evaluation scores, please check: Evaluation Details of Biocreative BEL Task
Examples can be found at: Example Submission
Please do some tests and let us know if anything is not clear. We are happy to consider modifications suggested by the participants, and we would be glad to provide further explanation either on this list or directly -- do not hesitate to contact us!
Temporary Limitations of the Interface
A new version has been released. This version has been improved in the following points:
- Detailed report about all true positives, false positives and false negatives in the submission
- The secondary scores (Fs, Rs) should now be correct
- Handling of functions which are equivalent to 'act'
- Equivalence of namespaces (HGNC is now the namespace used in place of HGNC, EGID and MGI in the gold standard; however, participants are free to use the namespace of their choice)
- Evaluation of placeholders
Please notice that there still a few limitations in the current version of the evaluation interface. We are currently working on solving them.
A new version overcoming these limitations will be released soon, but in the meantime, please be aware of the following shortcomings:
- The secondary scores (Fs, Rs) are not always completely correct. Besides the Ts score is still missing.
- If a BEL statement is syntactically malformed, presently the whole statement is ignored. This is the correct behavior, but in the upcoming version a warning be issued
In case you feel that something is missing in this list or something does not work as expected, please do not hesitate to send us a quick note: firstname.lastname@example.org