How To Setup

Zurück zur News-Liste

It can evaluate the following two tasks for building large-scale knowledge graphs:

Knowledge graph population. Given the name and type of a subject entity, (e.g., a company) and a relation, (e.g., CEO) participants are expected to provide the value(s) for the relation. Here Gerbil SW will compare two graphs (the gold and system answer) and will calculate f1-measure, recall and precision.

Knowledge graph validation. Given a statement about an entity, e.g., the CEO of a company, participants are expected to provide an assessment about the correctness of the statement.

Here Gerbil SW will create a ROC Curve and calculate the Area under Curve (AUC).

Each statement must have a truth value predicate, which indicates how confident the system is that the statement is correct by using a value between 0 and 1.

If the system is 100% sure the statement is wrong it will give the statement a 0 truth value, if it is 100% sure the statement is true, it gives a 1 and a 0.5 indicates that the system doesn’t know.

Gerbil SW sorts the answers numerically starting with the statements which were annotated with 1.

Then the system answers will be checked against the gold answers. If the statement is true, the ROC will go up. If the statement is wrong the ROC will move to the right.

Gerbil SW provides a simple submission form as well as Leaderboards for each Task and Dataset.

This setup will explain first how to initialize and start Gerbil SW and, in the following, how to add your own dataset.

The first step to setup Gerbil SW locally is to clone the code from github:

$ git clone -b SWChallenge github.com/dice-group/gerbil.git

Secondly, change to the gerbil repository:

$ cd gerbil/

now initialize gerbil SW by using either

./init.sh or mvn install

This will generate the jar files of the Gerbil SW modules and will install them into your maven repo.

To finally start Gerbil SW you can simply run the start script

./start.sh

Add your own dataset

You can add your own dataset by creating the following folder structure:

$ mkdir gerbil.web/gerbil_data/datasets/

Now copy your files into the `datasets` folder:

$ cp /path/to/your/file.nt gerbil.web/gerbil_data/datasets/

Now we will add the file.nt to Gerbil SW. Open the datasets.properties file as follows:

nano gerbil.web/src/main/properties/datasets.properties

You can now see 4 datasets.

Delete all of them. (Make sure you use the Semantic Web Challenge 2017 data)

To add your dataset you must add the following lines:

org.aksw.gerbil.datasets.MyDatasetConfig.rdfFile=${org.aksw.gerbil.DataPath}/datasets/file.nt
org.aksw.gerbil.datasets.definition.MyDataset.name=My Dataset
org.aksw.gerbil.datasets.definition.MyDataset.class=org.aksw.gerbil.dataset.impl.sw. FileBasedRDFDataset
org.aksw.gerbil.datasets.definition.MyDataset.cacheable=true

# Change the following value to SWC2 if you want to use the dataset as Task 2 (Knowledge graph validation)

org.aksw.gerbil.datasets.definition.MyDataset.experimentType=SWC1
org.aksw.gerbil.datasets.definition.MyDataset.constructorArgs=${org.aksw.gerbil.datasets.MyDatasetConfig.rdfFile},${org.aksw.gerbil.datasets.definition.MyDataset.name}

Task 1 specific setup

You can specify which predicates should be compared by changing the value of the `org.aksw.gerbil.modelcomparator.properties` key in the `gerbil.web/src/main/properties/gerbil.properties` file. Put all predicates as the value (comma seperated).

For example:

org.aksw.gerbil.modelcomparator.properties=http://example.com/prop1, example.com/prop2

If your dataset file or the System file has a property `http://example.com/prop3` it will be ignored and will not affect the score at all.

Task 2 specific setup

You can specify the predicate which should be used as the truth value by changing the key `org.aksw.gerbil.evaluator.roc.truthProperty` in the `gerbil.web/src/main/properties/gerbil.properties` file.

For example

org.aksw.gerbil.evaluator.roc.truthProperty=http://example.org/hasTruthValue