I hear all the time about builders who are willing to cut a real estate agent out of a deal to "save" their clients a 3% commission. There are a couple of reasons this should concern you as the buyer.
Is it fair to the agent?
In some circumstances, this is simply unethical. If your real estate agent introduced you to the builder, then the builder should be thanking the agent. The builder would have had no opportunity to earn your business if the agent hadn't connected you.
The entire process results in a net gain for the builder, even if he pays the real estate agent a fair commission, because the builder wouldn't be building your house at all if the agent hadn't introduced you.
The agent honored the builder by extending the trust you've already placed in them to the builder—the very least the builder can do is return that honor by compensating the agent for the work and the extension of trust. If the builder has so little respect for that kind of professional trust and courtesy, you have to wonder whether he has any respect for his own profession and the associated trade—or for you, the customer.
Is it fair to you?
The other problem with this situation is that it's borderline dishonest. It doesn't bode well for this builder's future interactions with you. Think about this: if you're talking about a custom home on your land, you probably haven't yet defined the terms well enough this early in the conversation to know what the final price is going to be anyway.
At this early stage, the builder can tell you he's reducing the price by 3%, but you have no way of knowing. If you don't know exactly what your custom home plan covers, any price the builder gives you is an estimate based on what he tends to build for other people. It's easy for a builder to tell you he's shaving 3% off your home price if you haven't worked out together exactly what needs to go into your home and how much that will cost.
If you suspect this lack of honesty early in the process, would you really trust this guy to build your home?
There are many builders out there who respect the fact that real estate agents work hard to build a reputation, earn trust, and earn their clients' business through referrals. Earning that business is only possible through hard work, diligence, and consistently serving each client's best interests. A builder who recognizes and respects that work ethic probably shares that ethic himself, and he will serve you (and your agent) well.