Host overhead is killing your inference efficiency