GSoC '26 Week 4: The Ontological Guardrail and SPARQL Verification

The math blends (80/20 and 50/50) were working great, but the pipeline still failed on edge cases like mapping “George Lucas” to the historical figure “George W. Lucas”. Because of that we came to a conclusion that we also need a validation layer in the end, to catch these statistical anomalies. We can’t completely rely on the final output generated by steps 2 and 3.

For implementing that I came up with the idea of using domain and range of the entities. Every DBpedia URI has a few properties, the domain and range, we would compare the domain and range of subject and object, and see if they match. And we will also compare this to the sentence, to see if there’s any similarity. Based on this we apply penalty to the scores generated by earlier steps. If domain and range match, we approve the triple and apply no penalty. If there’s a type mismatch then we penalise the triple by -0.25 points from the score given by step 2 and 3.

This was working well for a while but there’s one very interesting observation I came across. Many URIs have the properties tags and using them we are able to penalise or approve the entities. But some entities don’t have these tags! Their properties are missing! When the model tries to find the properties and it can’t find them, then it simply applies a -0.25 penalty to it.

This is exactly what happened in the George Lucas example. The real George Lucas that was mentioned in the sentence, the tags were missing in its properties, so the model slapped a -0.25 penalty to it. And George W. Lucas, who is a historical figure and has nothing to do with this sentence, maybe had the tags in it like “person”, and because of that the model approved it.

And it received a higher ranking than the actual George Lucas who got penalised. The thing we observed is, sometimes an entity might be the correct one but it can have missing tags, and because of this even after being the correct one, it will face a -0.25 penalty.

The solution to this is applying a softer penalty to missing data, a penalty of -0.10. We are still penalising it, but not as harshly as a mismatch of tags.

And the final layer of the process will be the SPARQL layer, it’ll act as a final sanity check, it’ll check if this final triple given by layer 4 exists in DBpedia or if it is a new triple.

This approach fixed a lot of things, we correctly solved the George Lucas example and not only that, this approach was able to rightly generate triples from sentences.

See this output snippet:

In our weekly call my mentors suggested me to use an LLM in the validation layer, which will validate the final triple just like a human. Which really made a lot of sense because there are edge cases where math is doing its job but we still might get wrong outputs, for those cases we need the LLM to verify the final triple.

The plan is to feed it the input sentence and the list of top 5 triples generated by steps 2-4. Then it will analyse them and validate if the final triple is correct and makes sense logically. This makes sure that we only give out correct triples.

One more thing that was advised to me by the mentors was to test different vector value combinations, right now we are using 80/20 vector and lexical proportions in step 2 and then penalty system of -0.25 and -0.10. But why only 80/20, why not 70/30? Did you test the model with 70/30 or 90/10? What was so magical about 80/20?

We need to come up with a study that gives us the accuracy or the model performance with different vector values, so that we prove why we chose those specific values.

So that will be our task for the next week, using the LLM for final validation and testing the model on different sentences. The goal will be to give out correct triples for as many sentences as possible.