Perhaps you need to think less, many regular logic families
will produce this kind of risetime just by loading.
Now if you want FM "modulation shape" fidelity you're on a
different sort of hook. But if all it wants is "there to here"
frequency sweep / chirp, maybe it's easier than you think.
The narrowest chirp you could get will be from an advanced
CMOS logic family, make a one-gate-delay positive-edge-only
"yip" with a quad NAND gate. Burden the output (incl the
varactor) with external C until you like the risetime (if too fast).
If too slow, pick another logic family.
A very linear ramp would need a fancier circuit. Like maybe a
current source charging a C, with a shunt switch to reset / hold
off driven by a suitably-matched one-shot.