count extra inner references if inner_reference is non-0, and similar for outer references. This is because the handle table might have another reference which foiled the overall logic.