compare is #define'd in inet/generic/assert.h:
#define compare(a,t,b) (!((a) t (b)) ? bad_compare(this_file, __LINE__, \
(a), #a " " #t " " #b, (b)) : (void) 0)
and bad_compare() is defined in inet/inet.c.
If the relationship between the 3 arguments in compare() does not hold, some debugging output is emitted and then Minix is terminated.
For example, if compare(result, >=, 0) is called and result (the first argument) is -1, Minix will be terminated.