Question about benchmarks

[marfl]

Nice project and impressive results!

I believe the current benchmark comparison with fbas_analyzer is slightly misleading when it comes to splitting sets and blocking sets. AFAIU python-fbas currently terminates once it finds a minimal set with minimal cardinality, whereas fbas_analyzer attempts to find and output all minimal sets. That seems to me to be a harder task, which distorts the comparison slightly. (Admittedly fbas_analyzer doesn't seem to support anything that would be better suited for comparison here...)

The results for quorum intersection by themselves seem to be good evidence already that your approach here is better suited for larger and more complex FBASs 👍

[nano-o]

Yes you are right, I should clarify that. I had in mind that only the minimum of the cardinality is what users would be interested in (to compute some kind of Nakamoto coefficient). Also, there may be exponentially many minimal splitting sets, so listing them is bound to be a problem as network size grows. For a fair comparison, would it be easy to add e.g. a command to compute the minimum cardinality of splitting sets to fbas_analyzer?

Thanks for checking it out!

[marfl]

I had in mind that only the minimum of the cardinality is what users would be interested in (to compute some kind of Nakamoto coefficient). Also, there may be exponentially many minimal splitting sets, so listing them is bound to be a problem as network size grows.

Yep, fair points!

For a fair comparison, would it be easy to add e.g. a command to compute the minimum cardinality of splitting sets to fbas_analyzer?

I played around with such ideas in earlier versions but the way it's wired up at the moment it would not be completely trivial to add. Off the top of my head, the way to go would probably be to rewrite the analysis functions to (optionally?) prune search paths based on the smallest-cardinality set already found. In any case I won't be able to be of much help personally as the time I'm able to invest into fbas_analyzer these days is nearly 0.

So it seems fair to state that fbas_analyzer is limited in this way in its current implementation, i.e., that if you're only interested in the smallest-cardinality sets then you're still forced to run the full enumeration.

In addition you could also add top tier determination to the benchmark; here you have equal functionality between the tools. (Just an idea!)