Qt is a custom-drawn widgetset, it is not supposed to be native. It just kind of immitates native, but not perfectly.
But all square buttons in LCL-Carbon are 100% native, that's just how Aqua draws buttons. If they are small they are round, if they are bigger then X pixels then they are square.
But Qt is 32/64bit, so there is a way to use it for make a 64bit Lazarus that creare 64bit applications?
You don't need a 64-bits Lazarus to create 64-bits applications. But in general yes, you should be able to obtain a 64-bits Lazarus with Carbon, but as you tried you have found a compilation error. It seams that part of the IDE supposes that Carbon is used under Mac OS X, looks like a bug.
And is there a way to use 'appkit' under Qt?
This is a confusing question. How do you want to use it? Appkit is part of the Cocoa framework, which is used by Qt.